REST API, intermediate proxies and the Vary header

Development Developing for Plone
,
1

we've found an issue recently while developing an add-on for one of our customers.

our add-on is an agenda that includes days and appointments; for days we implemented a view that shows all appointments on that day and a calendar to make navigating the agenda easy.

the calendar is build on the fly using application/json calls to other days on the same month to mark all days with appointments and to make fast updates when a user selects another day.

resuming: if I traverse to the default view of a day, I get normal HTML; if I request application/json using an AJAX call to the same default view, I get JSON.

everything was more or less working until we deployed it and then we discovered that Varnish was caching only one of the versions breaking the whole thing.

enters the Vary header; according to Varnish documentation:

If the origin server sends Vary in a response, Varnish does not use this response to satisfy a later request unless the later request has the same values for the listed fields in Vary as the original request. As a consequence, Vary expands the cache key required to match a new request to the stored cache entry.

so, according to this I can fix the issue adding the Vary header to my response.

my next step was testing the following on our code:

    def __call__(self):
        self.setup()
        self.request.response.appendHeader('Vary', 'Accept')
        return self.index()

and it works fine, except for one detail: it's only applied to the HTML response.

does anybody else has found this issue? how do you deal with it?

I'm wondering why this header is not included by default in plone.restapi (any comments, @tisto?)

tomorrow I'm going to test a workaround using plone.app.caching; I'll post my results after that.

2

Let me see if I understood correctly: when you make a request to your view with Accept: text/html then you get a response with the Vary: Accept header. However if you make a request with Accept: application/json then the response does not include the Vary header. Is that the issue?

Is plone.app.caching enabled in your tests? Since this add-on sets the Vary header itself it may be messing things up. I would try first testing with no plone.app.caching and no Varnish involved. Then work from there.

If you could post some examples using curl or httpie (wonderful tool btw) maybe I can spot something.

3

no, I'm adding the header manually in our code; but I would need something similar on plone.restapi.

if I understand correclty, plone.restapi patches the Zope server and intercepts request that have the Accept: application/json header on it.

configuring this in plone.app.caching seems the way to go, but it's also complicated so we have decide to drop the use of plone.restapi in favor of our own JSON view.

I think this problem is going to be more evident as soon as usage of plone.restapi increases.

this could be even used as a vector to attack a site if the administrators don't know what they are doing.

it could be important to document workaround in plone.restapi documentation, but I have no further ideas and we have a deadline soon.

4

Just for the record, plone.rest is the package that does the content negotiation. Please create an issue on the plone.rest issue tracker describing the problem in detail and with HTTP examples to reproduce the problem. We might be able to look into this next week at the Beethoven sprint. Though, we need more details for sure.

1 Like
5

thanks, I have opened a new issue as you suggested; let me know if we can help you on this:

6

I am so glad to see this is being worked on. We are experiencing this issue but luckily we caught it in testing.

Current status looks like the Vary header will NOT be the answer to the cache poisoning in varnish, and the approach is to construct a unique url to distinguish application/json requests from text/html requests (and others)

Thank you @jensens and @tisto for REST traverser ++api++ by jensens · Pull Request #113 · plone/plone.rest · GitHub

2 Likes