Plone page composition and ESI

Some might remember that when Deco was introduced, possibly the main reason for the chosen "HTML based" page composition was that page fragments could be cached individually simply with HTTP caching headers and the final page could be composed outside Plone (possibly requesting non-cached fragments in parallel). The main technology for that back then and AFAIK still is ESI with Varnish.

Have anyone actually tried that? (In addition to me.) And how important do you feel that approach for Plone in the long term? Do we still believe in the "Blocks based layouts" as visioned in 2009? Or are we abandoning that in favour of headless CMS and browser rendered UI?

Personally, I've counted a lot on that, and during this year most of our university sites will update to HTML-based site layouts with ESI-cached tiles.

  • some of my reasons are theming related: our layouts are theme specific and we use only Diazo to transforms things like breadcrumbs and portlets to fit our theme syntax

  • some are Mosaic: WYSIWYG editing of more complex layouts and HTML based content views

  • and some are for performance: subrequest based Mosaic default composition gets slow very fast, but with ESI, multiple instances can render tiles in parallel, and with some collaboration with Varnish, we can cache tiles safely even for logged-in users

In short, because it's hard to make Plone render complete pages (with all content, viewlets, portlets, tiles and diazo rules) fast, I count on splitting rendering in to small pieces and slowly optimize those pieces one piece at time.

1 Like

It's not really clear what your question is.

You seem to be suggesting that we change mosaic to make ESI compulsory rather than optional? That would be very bad. Plone is already complex enough to setup. Finding a way to optimise subrequests to make this performant seems possible to me and wiser. (same applies for postgres... I've no idea why that is even being considered).

I was afraid to ask that so straight. I'll rephrase.

Subrequests are implemented the way they are to keep them isolated. They can be optimized by breaking that isolation, but that would defeat their purpose. And I'm unaware of better solutions than ESI to parallelize fragment rendering. Mosaic is very slow for logged-in users without that.

Newtdb has still a lot of open questions, but with Plone 6 we become compatible and it has kind of a promise to fix performance issues we have because of ZCatalog. But that's a very long timeframe.

Depending on an external service for implementing a core functionality would not be acceptable. It will conflict with setups using something else other than Varnish. I would require to change the varnish configuration - perhaps you need root permission or administrators help...sorry, this is not feasible and not desirable.


Thanks. I rephrased the questions to focus on others' experience on ESI and if they still feel it's relevant.

I think retaining a html frontend for content is very important (maybe not for the editing interface). CMS's are primarly about content and I don't see the web being change to REST and headless except for webapps or some people who want to spend too much money making things fly around. HTML is accessible and universal.

I'm not the right person to ask about ESI and blocks but I can see its a useful thing to have up our sleeve. If its not too fiddly to setup it gives a way to parallelism the page composition. It would be even nicer if we can move that parallelism all the way back to the browser via something similar to facebooks bigpipe.

But thats for complex sites that really need the speed. Our story should be something that works out of the box, as fast as we can make it, that still retains ways to make it faster if you want to invest in it.

What I think you seem to be asking is this: Mosaic could be must faster if we ditch subrequests and ESI. But I'm still not sure we've investigated all the ways we can make it faster while retaining subrequests and ESI. For example, one optimsation might be to compile the page so if its composed locally a bunch of work is already done. Or we have two apis to call a tile, one using a request type api and the other a faster one thats used in of a subrequest doesn't have the overhead of forcing all the data in and out of a request. Or perhaps there is caching that can be done to reuse requests between subrequests? Do we have an idea where the time is being spent with the subrequests and mosaic?

We used ESI 15 years ago at Zope Corporation (by funding the ESI support in Squid). It never became mainstream, it never became productive. From my point of view: dead technology and wasted time. We're moving towards headless system, towards client-side systems....I do not see any real usecase for ESI in this decade.


1 Like

AFAIK bigpipe is a single long streaming HTTP request, which streams the layout in HTML at first, following a bunch of script-tags injecting the content into holes in layout. Only that those tags can come in any order from the backend because of parallel rendering. I believe, it requires very similar to the current design with ESI tiles.

Yet, I don't see its benefits compared to doing ESI-like composition in browser? It's a bit faster, because browser don't need to ask for the tiles?

I don't have any single reason for the bad performance. Small things in traverse, security, recursive groups support and dexterity's magical getattr quickly add up when they are multiplied with tens.

But once we replace subrequests with multiadapter calls we are almost back where we started: with previous viewlet, portlet and content based composition products. Then we should just use those and drop whole Mosaic stack and just write the new editor to use e.g. page specific portlet manager with static text portlets instead. (That was an alternative @sneridagh experimented before Barcelona Mosaic sprint.)

I'd like to hear more discussion about


1 Like

@zopyx this is a specific use of esi for speeding up mosic rendering. The question wasn't do you like esi in general.

@datakurre the difference is in perceived responsiveness of the page some of the page is loaded quick. Composition at varnish still has to wait for all the bits to finish before sending to the browser so the user might still wait 2sec to start to see anything render.
But bigpipe isn't that important. Towards your original question, I think the architecture of mosaic being tiles can can be composed outside of plone seems powerful and it would be a shame to lose it. Bigpipe is just another way that same architecture can be used to get a better result for viewers given the particular usecase. I dont think its a good idea for all themes. And neither would relying on esi. But good to know both are possible.

@datakurre so why can't we have two ways to render a tile? Adapter call and via external request. As long as we keep them equivilent (not extra info passed via adapters) then we can keep external request api for free right?

we have never used ESI even as all collective.cover tiles inherit from the ESI one on plone.tiles; we have never experienced performance issues neither, but maybe because our use cases are mostly anonymous users and the current way we cache things is good enough.

in this case you have to make a better analysis to isolate the chief cause of the performance issues, then you can work on that and make some tests to measure the gains.

why we can't have 3 or 4 ways? we have an issue in the Plone community: instead of solving problems we like to create new ones.

1 Like

This is wrong. Varnish doesn't do that with ESI by default. With ESI enabled the default behavior of Varnish (using 4.1.4 currently) is to start streaming page immediately (causing stops when it's rendering).

Yes. I've been doing this and already optimized a lot of things in our pas.plugins.ldap fork, disabled RecursiveGroupsPlugin and removed Python getattr implementation from my Dexterity. After those, there was no longer a single major issue visible in profiling. Right now I'm happy, but probably go back to profiler to look the next issue later :slight_smile:

I think, Andreas' point is important, because with headless Plone being the future, there's not much point in continuing the development of the current HTML based Mosaic :slight_smile:

FWIW, we've been rendering tiles without subrequests successfully for a while now:

So, we already have four different ways to render tiles :slight_smile:

  • Direct request to tile as done by Mosaic Editor

  • ESI request to tile, which could have additional ESI related headers (I do that to cache ESI tiles separately in Varnish)

  • Plone subrequest to tile, which could be the same as above, but also allows tiles to use state information from parent requests (already done by viewlet tiles)

  • Multi adapter call to tile... not sure if Castle has tiles relying on that :slight_smile:

@datakurre I'm not sure your point right now.

Tiles are essentially views. There is really only 2 ways, multiple adapter or requesting url. What you're talking about is page composition. In that right now there are 2 ways: inline(normal) or esi.

I'd be less concerned about how something is technically achieved. The reason we did away with subrequest in castle was for performance reasons.

But once we replace subrequests with multiadapter calls we are almost back where we started: with previous viewlet, portlet and content based composition products. Then we should just use those and drop whole Mosaic stack and just write the new editor to use e.g. page specific portlet manager with static text portlets instead.

This statement doesn't make sense to me. I wasn't aware that blocks was only designed to be used with ESI and/or parallel rendering of layout. You're getting too hung up on the mechanism used to accomplish this. The multiadapter variant is just an optimization. Go ahead and use ESI.

@vangheem I was not truly serious in the latest. Sorry.

I don't truly think it's wrong as such that we have multiple alternative composition methods for layouts and tiles. The multiadapter based transformchain even makes it easy to override customize transforms. But there's real danger that we start relying on the special features of the method of our choice, making tiles behave differently or even being incompatible with different rendering methods.

Or should we be honest and explicitly support multiple different methods at the same time in a way that different tiles could be designed and optimized for different methods? But that would make it conceptually more complex. So, the current state is a good compromise?

That said, I really thought that subrequests were just a temporary solution until we had something like ESI working well enough, and because of that, going back to multiadapters feels like a step backwards for me. Sure I'm happy, if we can continue to have the both ways.