Plone.recipe.varnish and purge

yurj · December 7, 2021, 4:38pm

plone.recipe.varnish produce this code:

sub vcl_recv {

    set req.backend_hint = balancer_0.backend();

    if (req.method == "PURGE") {
        # Not from an allowed IP? Then die with an error.
        if (!client.ip ~ list_purge) {
            return (synth(405, "This IP is not allowed to send PURGE requests."));
        }
        return (hash);

shouldn't be return(purge)? with hash, the content isn't purged until 1 minute when I save the page. With purge, is immediately purged and fresh content is served.

fredvd · December 7, 2021, 4:56pm

That's strange: the jinja template in plone.recipe.varnish lists return(purge); and that line hasn't been touched since 4 years:

github.com

collective/plone.recipe.varnish/blob/07b6242d00366018dfe0c604c8f343f10ad02240/plone/recipe/varnish/templates/varnish6.vcl.jinja2#L93-L98

    
      
          if (req.method == "PURGE") {
              # Not from an allowed IP? Then die with an error.
              if (!client.ip ~ list_purge) {
                  return (synth(405, "This IP is not allowed to send PURGE requests."));
              }
              return(purge);

yurj · December 7, 2021, 6:41pm

I'll check tomorrow, really strange!

yurj · December 8, 2021, 9:30am

this is exactlythe file that is in the egg wheel:

if (req.method == "PURGE") {
    # Not from an allowed IP? Then die with an error.
    if (!client.ip ~ list_purge) {
        return (synth(405, "This IP is not allowed to send PURGE requests."));
    }
    {% if minor_version > 1 %} 
    return (hash);
    {% else %}
    return(purge);
    {% endif %}
}
Archive: plone.recipe.varnish-2.3.0-py3-none-any.zip
inflating: plone.recipe.varnish-2.3.0-py3.9-nspkg.pth
[...]
inflating: plone/recipe/varnish/templates/varnish6.vcl.jinja2

in 28 Jan 2020 you removed this check. I think I've also hit this problem:

github.com/collective/plone.recipe.varnish

vcl for varnish 6 gets vmod_purge from wrong minor_version and 'hidden' purge

opened 07:41PM - 03 Sep 19 UTC

closed 08:56PM - 06 Oct 20 UTC

fredvd

@cleberjsantos I'm a bit confused by the new vmod_purge functionality you added …in the varnish6.vcl.jinja template at https://github.com/collective/plone.recipe.varnish/blob/master/plone/recipe/varnish/templates/varnish6.vcl.jinja2 First, the minor_version in the jinja template is that last number from the Actual varnish version and we have 2 and 3 digit varnish version, so the vmod_purge snippets are not activated when you think they are (on varnish 6.0.3, but not on varnish 6.1) https://github.com/collective/plone.recipe.varnish/blob/7e34dca18a75a54e3430c16745ccd25b48ee9362/plone/recipe/varnish/recipe.py#L284 Second: with the vmod_purge activated there's now a call to custom_purge from vcl_hit and vcl_miss subroutines. from the generated varnish.vcl: ``` sub custom_purge { if (req.method == "PURGE") { set req.http.purged = purge.hard(); } else { set req.http.purged = purge.soft(); } if (req.http.purged == "0") { return (synth(404, "Not Found")); } else { return (synth(200, "Purged")); } } ``` When I `curl -XPURGE http://varnish:purgeport/path/to/item` for an item which is not already requested and in the cache I get the synth 404 as a reply: ``` <?xml version="1.0" encoding="utf-8"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html> <head> <title>Varnish cache server: 404 Not Found </title> </head> <body> <h1>Error 404 Not Found</h1> <p>Not Found</p> <h3>Guru Meditation:</h3> <p>XID: 32775</p> <hr> <p>Varnish cache server</p> </body> </html> ``` But when the item was indeed in the cache, I get the full html of the item delivered with status 200: `return (synth(200, "Purged"));` from custom_purge seems never to be executed. I think I understand the reasoning behind the new routine: instead of evicing the item from cache where the next client request will have to fetch it from the backend, you load a new fresh version directly into the cache, ready for the next client request. But it's very hard to track a correct purge now, it took me quite a search to understand why I wasn't seeing a MISS the first time after doing a PURGE. And I'm still digging to verify the purge is actually working, which is a bit difficult for the current testing set up where I'm not easily authenticated to the Plone backend without going through the whole stack, there's no header or anything else indicating the purge actually happened.

because recipe.py got 0 as the minor_version (in the buildout there's varnish_version = 6).

I removed the cached wheel and rerun buildout but I still get that template with the minor_version. Here:
https://pypi.org/project/plone.recipe.varnish/#files
there's the py2 version and the tar.gz which have the same old template. Buildout downloaded the .tar.gz (bin/buildout -vvv):

Installing 'plone.recipe.varnish'.
We have no distributions for plone.recipe.varnish that satisfies 'plone.recipe.varnish'.
Getting distribution for 'plone.recipe.varnish'.
Fetching plone.recipe.varnish 2.3.0 from: https://files.pythonhosted.org/packages/08/af/28e0474501bda52f761c5d8c794b69eda5accd6180b223fcd00d0129c9e3/plone.recipe.varnish-2.3.0.tar.gz#sha256=172246bbee55f1379200a32ea21d10ddc1ddb7c98ec30e855a579aef73282b7e
".../bin/python3.9" "-c" "import sys; sys.path[0:0] = []; from pip.__main__ import _main; _main()" "install" "--no-deps" "-t" ".../buildout-cache/eggs/tmp1sadhayc" "-v" ".../buildout-cache/downloads/dist/plone.recipe.varnish-2.3.0.tar.gz"

maybe a release is missing? Or is it better to git clone the recipe?

fredvd · December 8, 2021, 11:30am

I've just released a 6.0.0b4 with updated default download location for varnish 6.0.9 LTS that was released at the end of november.

If nothing get's reported I'll re-release it as 6.0.9 final at the end of the week. Then at least that version will pop up as a final release and not the by now ancient 2.3.X release.

The reason the 6.0 stayed so long in beta is because a lot of cleanup was done: a lot of switches to support Varnish 4 to 5 to 6 in different templates with all these conditions were getting too complex to maintain. Around that time Varnish also came with the LTS version.

After that they switched to a half yearly release cycle. But so far none of those intermediate versions has been designated as a new LTS by the Varnish organisation. Maybe the 7.0 will be. Then we could take a look at updating the recipe and releasing plone.recipe.varnish 7.0.X

Varnish drops security support after max a year on every non LTS release, it was too much work for the current recipe maintainers to handle those for the short period that Varnish version is actually supported.

Using Varnish releases above 6.0.X with the generated varnish.vcl recipe is 'unsupported' . And with unsupported I mean that it might seem to work fine, until you really test it rigorously by inspecting varnish_log on test requests and then discover that some small things did break and your Varnish doesn't really cache or caches less than expected. Been there, done that :-/ If you do want to to use 6.0.X < varnish <= 7.0, copy the generated varnish to a custom version and tinker away.

If someone in the Plone Community does want to support an intermediate release, one could create a 6.5 branch of the recipe on GitHub and release it under that recipe version (i..e. recipe semver = supported varnish semver). But Varnish 6.5 is not suported anymore with the release of 7.0.X.