How to best handle caching invalidation on folders and collections?

Continuing the discussion from Caching headers issue on collection:

We have being experiencing issues with caching invalidation on folders and collections mainly because the modified property of these items is associated with the last time the item itself was modified and not the last time some new item was added to a folder or some other new item should be displayed in the result of a collection query.

Because of this situation we end up with inconsistent results for users accessing the site behind a caching proxy (mostly, all anonymous users).

Also, I can see that even content depending on Last-Modified headers could easily become outdated when using portlets that display results of a collection.

Our current configuration on those items is something like this:

I'm aware of the ESI support in Varnish, but I have no previous experience implementing it and I don't know if our portlets support this feature (I know tiles do).

Lately I've been experimenting with the ETagheader to solve this issue, but I would like to know if there are some downsides (besides performance) on this approach or some alternatives.

As you can see I'm using the lastModified, catalogCounter and resourceRegistry tokens as described in our ETags documentation.

How do you deal with these kind of issues?

Can I use both Last-Modified and ETag headers at the same time?

The solution I use is normally to just cache for a short time (like 5min) which keeps it fresh enough while still enabling the ability to handle a huge influx of requests. This is especially true for homepages with collections on.
I haven't used the catalogcounter in an etag before but at a guess I'd say the only disadvantage is that if you have a lot of changing content unrelated to your collection then it will cause more invalidations and extra load and delays for users. But most sites don't get updated that much so it's probably fine.

One interesting approach I haven't tried out yet is - https://pypi.python.org/pypi/collective.purgebyid.
It doesn't state if its enabled for collections. I'd guess its not but this could possibly work for collections that don't change much. For example lets say its a news listing and you are showing the top 5 news items on your front page. purgebyid would mean that if anyone of those items change then your collection would also be purged. However this doesn't handle the purge you need to happen when a new news story is published.

The only way to handle purges of collections such that you don't have unneeded backend requests would somekind of event subscriber which runs every collection in a site on each content change to see if the collection results have changed and then doing a purge if it has. I'm not sure if anyone has created a plugin for this yet. But this could be computationally expensive if you have many collections but if you do it via async it might be a better tradeoff it have a lot more reads that writes and not many collections. There might be some optimisations you could make like instead of running each search in full, just work out if a given content object is satisfied by a given collections query by restricting each query to just that content object.

As you've discovered, Hector, last-modified validation is not sufficient for views the aggregate data from other content items.

I routinely use the catalog counter in an ETag (I think it's included in one of the standard plone.app.caching profiles) and it has worked well, though my sites may have lower traffic and/or editing rate than yours.

Dylan you have some good ideas about what to pursue in the case where that ETag isn't enough. Though just purging collections wouldn't be enough if there are also custom views that do catalog queries. Maybe a global mapping of catalog query to URL could be built up as pages are rendered, and then when an item is edited it could be matched against those queries to determine which URLs to purge.

Or maybe just a mapping of result UIDs to URL, and then you don't have complications in figuring out which queries are matched. Maybe do it as brains are accessed so you don't waste time and memory on the lazy results that aren't accessed. This should probably go in redis so you can purge pages that were rendered by a different instance.

@davisagli result uids aren't enough when new content is added.

But I think your idea of query to URL mapping is maybe not too hard to implement. It could be built up in transient memory like the query plan so u don't have write on read. Then u could either have it automatically run lots of queries on any content update or you could manually go into and Cherry pick which URLs u want purge requests for and save that purge plan to the zodb.

if you have a saved mapping like
{'portal_type':'NewsItem'} -> /news
Then if /foo/bar was updated you could do a query like
{'portal_type':'NewsItem','path':'/foo/bar'}
If non empty then /news needs a purge. Should be reasonably quick. Or better yet query using UIDs?
and with async support (liked delayed indexing) it won't slow down content saves.

The end result is a site you can cache any page you want with it always being up to date. Thats pretty cool.

you do so in Varnish? plone.app.caching doesn't seem to support that and I think it could be a good idea to implement this feature.

yes, the catalogCounter invalidates things too fast, but I have no other easy way to do it; I'm not convinced on the purge by id approach, caching for a short time seems easier.