Plone 5.2 - Python 3.6 - plone.recipe.varnish and Varnish 6.0.6

Hi,

I try to migrate a Plone site (used by a charity) from Python 2.7 - Plone 5.0 to Python 3 and Plone 5.2. I already migrated the used Plone add-ons for the site to Plone 5.2 and Python 3. I could run buildout for development with a single instance. I'm currently working on the buildout for the production environment. I was done with the instance, the zeoserver and the supervisor part of the buildout script. I could create a new zeoserver with four instances and start and stop it with supervisor.

I worked on the Varnish part of the script yet. I used the add-on plone.recipe.varnish for this. I tried with the latest release and Varnish 4.x first, but always got an error after running supervisor about purge. It stops at the line with 'import purge' and searches for a missing library in the Varnish build (the one for purging).

I changed my setting to use the sources of the recipe from the Github.com repository with Varnish 6.0.6 and the error disappeared. I could build Varnish from source. I could start supervisor, but varnish errored with the following entry in its log:

::
Error: Could not get socket 0.0.0.0:10001: Address already in use
(-? gives usage)

I looked with netstat for the used ports and got varnishd listening on that port. I opened my browser on this port (localhost:10001) and could view my Plone site. Although supervisor / varnish throw an error message to the log and the supervisor status of Varnish was:

::
varnish FATAL Exited too quickly (process log may have details)

I had a look into the 'var' directory of the buildout too and there is a cache folder with current content (py-files). Thus there is some caching once I added / edited some content to the site.

I don't know how to fix the issues with supervisor / Varnish. You could find the buildout script which I created and used for building the site on Github:

Because of issues with building haproxy (error with chmod 0755) I currently dropped the work on migrating this part of the current setup yet.

Kind regards,
Andreas

We are running plone.recipe.varnish 6.0.0b1 w8th Plone 5.2.1 and python 3.7 without issues.

If you also want to build haproxy, I created the python3 branch on github, it builds the latest haproxy and we are running it in production https://github.com/plone/plone.recipe.haproxy/tree/python3?files=1

Hi Mikel,

thanks for the hint. I was able to compile and install haproxy with your
branch. I could also run it with supervisor.
But the issue with varnish is already valid. I got the information that
varnish exited too quickly further although I was able to find varnishd
running with netstat on the expected ports (set for 'bind' and 'telnet').

I was also not able to get haproxy connected with varnish, using the
current vcl-file from the production enviroment (setup using a Python
2.7 and Pone 5 environment). This vcl-file is here:

And the buildout script is this (without haproxy and varnish configuration):

For the varnish/haproxy connection testing I changed the entry for the
vcl from 'varnish5.vcl' to 'varnish.vcl'.

Once I run buildout with this configuration I could start haproxy, but
it stated that it could not find a zope backend.

Kind regards,
Andreas

I have never used a custom varnish.vcl file, so I might not me helpful here. We are running varnish with the standard configuration file created by the plone.recipe.varnish recipe, here you have it: https://github.com/collective/plone.recipe.varnish/blob/master/plone/recipe/varnish/templates/varnish6.vcl.jinja2

We have the buildout configuration in a varnish.cfg file that looks like this:

https://pastebin.com/r7DbV4Zi

This folder is probably unrelated to Varnish but could be the Chameleon templating cache. Zope Page Templages are compiled and Chameleon has the option to persistently store the compiled results, also between Plone restarts.

Otherwise on every first template request the compiled result is stored in memory for the duration of the Plone process running if you are running in production mode. If you run in debug mode (to do development), every template is freshly compiled on every request.

plone.recipe.varnish 6.0.0b1 generates vcl which is only suitable for Varnish 6.0.6. IIRC the purge command got extracted in its own module in Varnish 6.

We had support for 3 different varnish versions in the recipe, but it got too complicated and became unmaintainable. Last year I found out purging was completely broken if you used Varnish 6.0.3 because snippets for an alternative purge strategy got activated based on a faulty programmed switch that confused the minor and patch version fed into the recipe on varnish-version.

Also Varnish the organisation changed ther release strategey because they waited for user feedback before every release, nothing came back, and as soon as the release was made people started complaining their Proxy server crashed. so they now release whatever is there every 6 months in a new major release and nominate a 'stable' version from existing versions every x years.

6.0.X is the current LTS. And stuff does break, Varnish 6.1, 6.2 and 6.3 each have small tweaks which kill the VCL we generate for 6.0LTS.

I 'announced' plone.recipe.varnish. 6.0.0b1 last month at the PloneTagung and asked who was using the recipe and for feedback . Turns out almost everybody 'knows' about the recipe and looked at the generated VCL at some moment for inspiration but then tweaks the vcl and switchsed to a custom setup. The curse of a 'generic' black box recipe, ah Well :wink:

This is my current buildout setup. I have switched in this project from haproxy to using the internal Varnish load balancer (Director). Down side is that you loose sticky sessions for logged in users, this is round robin, but the big advantage is that grace healthy and grace-sick are working properly.

Plone 4+ should work without sticky sessions but there have been edge cases independent of performance reasons: one I saw 5-6 years ago sporadically was an error on a newly created page: when you save your version on zeoclient1, zeoclients 2, 3, etc. need to get this information through the zeoserver connection. If the browser request for the view of the new object gets to zeoclient2 before the ZODB can sync it to the clients, the page does't exist yet :stuck_out_tongue:

I found an example for sticky sessions from a Varnish blog post but it is not trivial. See the TODO I created for more info in the repo: plone.recipe.varnish/TODO.rst at master · collective/plone.recipe.varnish · GitHub

[varnish-build]
recipe = plone.recipe.varnish:build
url = ${versions:varnish-url}

[varnish-configuration]
recipe = plone.recipe.varnish:configuration
backends =
       www.website.com:127.0.0.1:${conf:zeoclient}
       www.website.com:127.0.0.1:${conf:zeoclient2}
balancer = round_robin
grace-healthy = 60s
grace-sick = 600s
varnish_version = 6
vcl-version = 4.1


[varnishd]
recipe = plone.recipe.varnish:script
bind = 0.0.0.0:${conf:varnish}
cache-type = malloc
cache-size = ${conf:varnish-cache-size}
secret-file = ${varnish-secret:location}
mode = foreground
name = ${conf:ramlocation}/${conf:projectname}-varnish

[varnish-secret]
recipe = plone.recipe.command
location = ${buildout:directory}/var/varnish_secret
command =
    dd if=/dev/random of=${:location} count=1
    chmod 600 ${:location}

[varnish-clearcache]
recipe = collective.recipe.template
input = inline:
    #!/bin/bash
    ${varnish-build:location}/bin/varnishadm -n ${varnishd:name} "ban req.url ~ ."
    echo "sent ban req.url ~ . command to varnish"
output = ${buildout:directory}/bin/varnish-clearcache
mode = 755
3 Likes

Hello Fred,

thanks for your answer. This reminded me to take a closer look the
Varnish documentation. I need some further reading to get a more
educated picture of Varnish caching.
The Plone setup which I wanted to migrate to Python 3/Plone 5.2 has not
a very have load and is running in a virtual machine on a server. Thus
the setting with Varnish and HAProxy is maybe not really necessary.
There is no balancing between instances on different server etc.

Kind regards,
Andreas

Hi! Using this recipe (I had just to remove vcl-version = 4.1) with load balancing by varnish I got this numbers:

picking 1 client, directly (url_list file contains a list of all the resources in a page, emulating roughly a browser) :

$ siege -t 10S -c 10 -f /tmp/url_list8080.txt 
** SIEGE 4.0.4
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		         355 hits
Availability:		      100.00 %
Elapsed time:		        9.31 secs
Data transferred:	       28.23 MB
Response time:		        0.26 secs
Transaction rate:	       38.13 trans/sec
Throughput:		        3.03 MB/sec
Concurrency:		        9.89
Successful transactions:         355
Failed transactions:	           0
Longest transaction:	        1.49
Shortest transaction:	        0.05

with varnish and 1 client:

$ siege -t 10S -c 10 -f /tmp/url_list.txt 
** SIEGE 4.0.4
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		        4858 hits
Availability:		      100.00 %
Elapsed time:		        9.45 secs
Data transferred:	      468.27 MB
Response time:		        0.02 secs
Transaction rate:	      514.07 trans/sec
Throughput:		       49.55 MB/sec
Concurrency:		        9.90
Successful transactions:        4858
Failed transactions:	           0
Longest transaction:	        0.87
Shortest transaction:	        0.00

with Varnish and 2 clients in round robin:

$ siege -t 10S -c 10 -f /tmp/url_list.txt 
** SIEGE 4.0.4
** Preparing 10 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:		        5060 hits
Availability:		      100.00 %
Elapsed time:		        9.19 secs
Data transferred:	      486.06 MB
Response time:		        0.02 secs
Transaction rate:	      550.60 trans/sec
Throughput:		       52.89 MB/sec
Concurrency:		        9.84
Successful transactions:        5060
Failed transactions:	           0
Longest transaction:	        1.09
Shortest transaction:	        0.00

Note: "logged in Plone" numbers change because varnish is configure to don't cache when you're logged in Plone.