Could you describe in more detail, how does the service "slow down"? How much?Just the scripted upload gets slower at some point or also use of Plone with browser? If only upload, is it still slow, if you change the upload folder?
_---- Yes, you're right. The scripted upload goes slow but I haven't thought of using the web browser when the problem occurred. I've also tried changing the target folder and the HTTP 200 takes roughly about 1-3 seconds based on the delta time in my records._
And if it's the upload, would you know, which JSON api implementation you are using? "plone.restapi", "plone.jsonapi", or something else? And which Plone version are you using? (Plone does not ship with JSON api yet, so it has to be an add-on.)
----- Here are the version we're running:
Plone 4.3.8 (4312)
Python 2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)]
PIL 3.0.0 (Pillow)
I've used the API available on plone.jsonapi.routes
How big is your ZODB Data.fs? (If everything is as it should, Data.fs should be relatively small, because all files are stored as such in "blobstorage" next to "filestorage" (where Data.fs is).
Data.fs was at 42GB
BlobStorage folder was at 35GB
I'm still trying to understand, where the bottleneck really is. Is it ZODB, configuration, JSON api or Plone folder implementation.
One generic issue with that amount of pages in Plone is too small number as ZODB connection "cache-size". If you came find "zope.conf" for your Plone, you should be able to find it, set it to some very big number and restart the service to see if it has any effect. (You may be able to find connection cache usage stats with browser, if you find the Zope "Control Panel". Either at /manage or /aq_parent/manage depending of your configuration.)
Here's what we're using in the zope.conf
each clients are using
<zodb_db main> cache-size 3000000**
<zeoclient> cache-size 4096MB**