If you maintain (classic) Plone 5.2 sites with a Varnish caching server and you have recently upgraded to the most recent Plone 5.2.10.1 or Plone 5.2.10.2 releases, you could run into a very specific issue where the pages are still served, but all css styling is broken.
This happens when you update your Plone 5.2 site to 5.2.10.1 or 5.2.10.2.
You might not be impacted at all by this, but finding the root cause took us so much time and we have no idea how common our setups are, that we think a warning here on Community is necessary.
@mauritsvanrees and me have been debugging this for most of the Ffriday, we are not sure yet how widespread this and/or if Plone 6 is also involved, because it depends on the configuration of the whole request stack and we might for example have changed some caching parameters we're unaware off.
Another setting that contributes to the site breaking is that we add the "X-Content-Type-Options nosniff always" on our responses in nginx. Setting this header is part of many 'best practice' Nginx setup guides and necessary to tighten security to not let the browser execute injected javascript in served non javascript files.
Why this happens and what happens:
The Plone 5.2.10.1/2 releases have an updated Zope 4.8.X release that has more strict handling and checks on the Content-Type header in the responses.
-
With a 'default' plone.recipe.varnish setup and an nginx -> Varnish -> Zope stack, on the first request the site_theme.css is served from Plone, stored in the Varnish cache, and returned to the client. So far so good.
-
After the cache ttl (in our case 1800s), the next request for the site_theme.css is served from grace (if you set this in p.r.varnish) and Varnish does a 304 if-modified-since request to Zope. What has changed is that Zope now responds with a 304 not modified, but adds a Content-Type response of text/html. Older Zope versions would leave out the Content-Type header at all.
-
Varnish sees the 304 from Plone, no body (css) is returned from Plone, but Varnish now sees and updates the Content-Type header nometheless on the cached object and starts serving the theme.css (and also .js resources) as text/html instead of text/css.
-
The browser now receives the site_theme,css with a text/html mimetype and refuses to process the css because of the nosniff header mentioned above. You’re also likely see errors for separate js resources on browser console complaining about the mimetype mismatch.
The nastiest part of this bug: you will only start seeing this issue after upgrading and restarting Varnish and Plone AND after at least the TTL time (1800/3600) seconds and you do a force reload in your browser of the page so the browser cache is not hit. But depending on your site traffic that could take longer as well.
Our curent work around is to revert the site to Plone 5.2.10. Another fix might be to remove the nosniff header.
Maurits has started a PR if you're interested in where in the code this happens, but this is work in progress and it is friday evening.