How to cut Plone instances restart time by over 90%

Plone Support Deployment & hosting
, ,
1

Continuing the discussion from How to save and load the catalog query plan?:

Today I was trying to test if an increase in the zeo-client-cache-size had some impact in performance on one of our sites and the results were pretty amazing.

First, I discovered that I've been wrong all these years thinking that we had configured a persistent client cache on our ZEO clients; in fact that was not the case.

I was very lucky to stumble upon the ZEO documentation on client cache; by reading it, I started looking for a file with .zec extension and I was unable to find it; I then realized the zeo-client-client was missing directive from our configuration.

I manually set the client directive in the zope.conf and started noticing some changes on the instance behavior:

2017-08-08T15:08:24 INFO ZEO.cache created temporary cache file '<fdopen>'
2017-08-08T15:08:24 INFO ZEO.ClientStorage zeostorage No verification necessary -- empty cache
2017-08-09T09:53:18 INFO ZEO.cache created temporary cache file '<fdopen>'
2017-08-09T09:53:19 INFO ZEO.ClientStorage zeostorage No verification necessary -- empty cache
2017-08-10T10:16:22 INFO ZEO.cache created temporary cache file '<fdopen>'
2017-08-10T10:16:22 INFO ZEO.ClientStorage zeostorage No verification necessary -- empty cache
2017-08-10T13:02:46 INFO ZEO.cache created persistent cache file '/opt/plone/buildout/parts/instance1/var/zeoclient-1.zec'
2017-08-10T13:02:46 INFO ZEO.ClientStorage zeostorage No verification necessary -- empty cache

The first time I restarted the instance nothing seemed to have changed and it took almost 3 minutes to warm up the cache as usual:

curl -o /dev/null http://localhost:8081/Plone

The interesting part is when I tried a second restart 10 minutes later and the warm up took only 15 seconds to finish.

The logs were showing something different:

2017-08-10T13:15:11 INFO ZEO.cache reusing persistent cache file '/opt/plone/buildout/parts/instance1/var/zeoclient-1.zec'
2017-08-10T13:15:13 INFO ZEO.ClientStorage zeostorage No verification necessary (cache_tid up-to-date '\x03\xc1Q\x8b\x007\x8f"')

The monitoring graphs were really interesting also:

Selection_023.png510×570 22.6 KB


Selection_025.png550×600 31.6 KB

As you can see, not only the time was reduced but also the CPU consumption, network transfer and disk I/O utilization.

The first restart occurred as 13:02 and everything was business as usual: high CPU and a lot of network traffic and disk I/O rate. the second restart happens at 13:15 and is almost imperceptible on network traffic and disk I/O rate.

I can say also that the instance responsiveness to requests was very high since the very beginning.

This test was done on ZEO clients running on a different machine than the ZEO server, now I need to test what happens if the ZEO clients are locally attached.

2 Likes
2

Awesome!

3

thats pretty much what you should expect to see with zeo-client-cache enabled. It makes no difference to a warn instance but it makes a big difference to restarts since it should mostly have what it needs locally after a restart. Without it a restart has to use both CPU and network to pull all that zeo cache back into ram all the way from the zeo.

4

What was the difference with local zeo?

5

interestingly the result were pretty different:

first, the restarts time gains were not so spectacular: without cache restarts were taking around 90 seconds; with cache, 60 seconds.

second, the CPU usage was quite similar; I wonder why.

6

I think it's worth to open an issue (perhaps a PR) to address the "problem" in the documentation. It's precise but not user friendly. You set the size of the persistent cache and think you're done. But that is not the case.

Or better yet: make the recipe give a warning saying "You think your persistent cache is setup but it is not, you fool."

7

i also didn't find any .zec-files on my servers, is it possible managing this on P4.3 by buildout.cfg?
I did not find out how to do this by linked doc: [quote="hvelarde, post:1, topic:4688"]
ZEO documentation on client cache
[/quote]

8

you have to use the following option:

zeo-client-client
Set the persistent cache name that is used to construct the cache filenames. This enabled the ZEO cache to be persisted. Persistent cache files are disabled by default.

1 Like