1 VM with Nginx + HAProxy
1 VM with Zope
1 VM with ZODB
After a deployment I manually click through the data intensive pages to warm up the caches.
The internal app is very low traffic (50 users), but nevertheless quite important.
I plan to setup a second VM with a second Zope instance - as a failover and also as a possible way to debug production data.
After a restart/deployment, I'd like to avoid to manually click through my app on two Zope instances.
Is there anything you could recommend how to warm up caches automatically after restart/deployment?
I would recommend to use two (or more zope instances). This way you can restart one of the instances while the other instance is available to handle requests. You already use HAProxy, so the load balancer will see if one of the instances is down and stop sending requests.
You could automate your deployment and do a rolling restart of the instances with a warm up. For example:
- Restart 1st instance:
- Wait until instance is up
- Run script to warm up the cache
- Continue to 2th instance
To warmup the cache you could use a script which loads
sitemap.xml.gz (or a set of predefined URLs) and hit all or several pages. Using bash/curl/wget or python/requests it should be easy to crawl the sitemap or a similar file.
The Ansible Playbook's restart_script role contains examples of how to do most of this. Take a look at https://github.com/plone/ansible-playbook/blob/master/roles/restart_script/templates/restart_single_client.sh.j2 for the Jinja2 template of a script to take a single instance offline, restart it, warm its cache, and put it back online.
Thank you very much!
Especially in Steve's script there are some good starting points.
I will have to make up my mind how this will work between different virtual machines, and also I'd possibly have to think about how to authenticate (possibly via basic auth).
I won't be able to start working on this immediately, but I will report back then.
"zeo-client-client" could help, too:
Thinking aloud. It should be possible to warm the cache with the restapi.
Did you try to analyse what is the cache content? You could populate directly the cache with a script at startup, inserting the keys and values.
At the first thought it seems simpler to just call pages and let Plone do the work the first time.