How to warm up caches

Current setup:
1 VM with Nginx + HAProxy
1 VM with Zope
1 VM with ZODB

After a deployment I manually click through the data intensive pages to warm up the caches.

The internal app is very low traffic (50 users), but nevertheless quite important.

I plan to setup a second VM with a second Zope instance - as a failover and also as a possible way to debug production data.

After a restart/deployment, I'd like to avoid to manually click through my app on two Zope instances.

Is there anything you could recommend how to warm up caches automatically after restart/deployment?

Thank you!

I would recommend to use two (or more zope instances). This way you can restart one of the instances while the other instance is available to handle requests. You already use HAProxy, so the load balancer will see if one of the instances is down and stop sending requests.

You could automate your deployment and do a rolling restart of the instances with a warm up. For example:

  • Restart 1st instance:
  • Wait until instance is up
  • Run script to warm up the cache
  • Continue to 2th instance

To warmup the cache you could use a script which loads sitemap.xml.gz (or a set of predefined URLs) and hit all or several pages. Using bash/curl/wget or python/requests it should be easy to crawl the sitemap or a similar file.

1 Like

The Ansible Playbook's restart_script role contains examples of how to do most of this. Take a look at https://github.com/plone/ansible-playbook/blob/master/roles/restart_script/templates/restart_single_client.sh.j2 for the Jinja2 template of a script to take a single instance offline, restart it, warm its cache, and put it back online.

1 Like

Thank you very much!

Especially in Steve's script there are some good starting points.

I will have to make up my mind how this will work between different virtual machines, and also I'd possibly have to think about how to authenticate (possibly via basic auth).

I won't be able to start working on this immediately, but I will report back then.

Thanks again!

Plone Foundation Code of Conduct