Large-ish Plone 6 deployment / baselland.ch

Hustle and Bustle
1

TLDR version

We (webcloud7) launched our first big customer with Plone 6 classic as the backend and an independent Frontend developed by another company back in October. The frontend uses the RestAPI to get all its data.

URL: https://www.baselland.ch a Swiss canton.

The site has, on average 500K to 1M (peaked at 2Mio) hits per 24h and approx. 100k to 350k API (Plone) requests per 24h. 300 Users are creating content on it.

The backend and frontend runs behind Cloudflare, which we use for caching, WAF, etc.

Tech-Stack:

  • Kubernetes
  • Docker
  • Plone 6.0.13 (Will be 6.1 next week)
  • ZEO setup with longhorn
  • Elasticsearch
  • Redis / python-rq
  • Keycloak
  • Grafana/Loki (Dashboards / Alerts)
  • CircleCI

On top of Plone 6, we developed some Add-ons and customizations to suit the needs of our new product and the customers.

We will launch more customers on the new platform this year :tada:

If you are interested, you can keep reading for more detailed information :slight_smile:

Kubernetes Cluster:

This was built with the help of Six Feet Up and is optimized for Exoscale. The Helm charts can be applied to any other platform as well. We installed clusters on AWS, DigitalOcean, and Google for testing purposes.

Some features developed for our Backend):

  • wcs.samlauth: State-of-the-art SAML2 Auth plugin with detailed documentation. GitHub - webcloud7/wcs.samlauth
  • wcs.adminauth CAS 3 Version based auth services for admins: GitHub - webcloud7/wcs.adminauth
  • Contributed Async/Redis/RQ-Python integration for collective.elasticsearch
  • Configurable ReferenceWidget (from my old Plone days) made Plone 6/6.1 compatible
  • We have our own Page Builder GitHub - webcloud7/wcs.simplelayout - No official release
  • Further, not yet publicly available features (Ping me if there is an interest):
    • Optimizations for Plone Classic UI, so it is only accessible with a valid session. The Plone Classic UI is only there for authors. No anonymous access is possible.
    • Plone Classic MapWidget based on leaflet.js and OpenStreetMap. Stores GEOJson, which can be easily indexed in Elasticsearch. Support for plone.supermodel (Import/Export). Single/Multi Value. Points/Polygons/etc.
    • Generic Widget based on a JSONField’s schema (Only supports a minimal number of types so far.
    • Support importing JSONField with schema for plone.supermodel
    • Opening Hours Widget, which generates openingHours - Schema.org Property output
    • Internal Task/Review System
    • Cache invalidation Client for Cloudflare (Async / RQ Python / Redis)
    • Our own WorkingCopy/Staging Implementation.
    • Trash/Bin Implementation - Retains deleted items for 30 days
    • RestAPI Performance improvements.
    • Custom Workflows
    • RQ Python Jobs dashboard for Plone
    • Generic Async jobs and scheduled jobs with rq python
    • Custom Rules to make certain fields contain a value, or a certain value.
    • Minimal Matomo stats integration.
    • Tons of small tweaks to meet our customers demands.

Other interesting points:

  • End2End Tests with Testcafe Studio
  • All JS parts are written with VueJS

We cache everything with Cloudflare, and invalidate only necessary parts if there are changes. This results in being able to Cache over 90% of all requests.

Screenshot 2025-02-07 at 8.05.03 AM
Screenshot 2025-02-07 at 8.05.03 AM703Ă—523 35.9 KB

Due to the nature that Cloudflare no longer caches content that did not get access for a certain period, and we install frequent frontend/backend updates, in which cases we have to clear the entire cache. Something above the 90% mark is realistic.

This also means we need significantly fewer resources to serve the same number of requests.

API optimizations:
For performance reasons, We had to implement several improvements to the RestAPI. One reason is that Exoscale single core compute power is quite a bit slower than AWS, Google, etc. by a factor of 1.5 to 2, and another reason is that the Frontend team was able to construct slow queries accidentally.

Those are typical 24h on our System - The 95 percentile is about 200 to 250ms - Those are only RestAPI requests, including images and files almost always with fullobjects and include_items. The API is not cached on Cloudflare for the frontend.

Screenshot 2025-02-07 at 8.31.24 AM
Screenshot 2025-02-07 at 8.31.24 AM1640Ă—1107 163 KB

14 Likes
2

@maethu congrats! This sounds like an exciting and challenging project! Could you imagine submitting a success story for plone.org?

We could also mention this in a news item on plone.de.

1 Like
3

Thanks!! I just submitted a success story! Getting mentioned on plone.de sounds great as well!

1 Like
4

I am curious about how is this implemented. Could you give us a general idea?

5

This is some impressive work!

I am curious about how many people were involved in this project, and for how long.

Congrats!

6

@maethu I'm wondering why you choose to not use volto for blocks and layout, ie admin UI? is it because you had a different layout model in mind or because it would have looked too different from the final frontend, or perhaps another reason?

7

Yeah sure. I'm also happy to share all the code in the future. The project is just not there yet to just be public :slight_smile:

  1. Create a dedicated "Trash" area with custom workflow, for dedicated access.
  2. This area is basically a singleton trash_content.py · GitHub
  3. Use "_get_id" method on container to generate ids for objects added to the trash
  4. Instead of deleting, move items to the trash. Patched manage_delObjects.
  5. Recursively Remove View permission and Index allowedRolesAndUsers with no value. This prevents it from showing up in the catalog queries. Catalog does not need to be patched (that was a big one)
  6. Add some annotations to the trashed content.
  7. Adapted some plone core features, like preventing trashed items from showing up on link integrity checks and relations fields in general. Disallow traversing on trashed content, etc.
  8. Restore content.
  9. Cron job, which deletes older objects, runs every 24h
  10. There is some more, like I don't put large structures into the trash because it takes too long, restapi endpoints, etc.

Here is most of the logic: trasher.py · GitHub

Happy to help implement a collective package if there is an interest.

2 Likes
8

Thanks so much!!

I can only speak from a technical side. There were many people involved from the customer side. My business partner and our project manager handled that part.

Infrastructure: Six Feet Up Implemented the Kubernetes Cluster for us.
Frontend: External company. Not sure about their resources, I had contact with 4 People (2 Devs, 1 QA, CEO)
Backend (Code, Docker, Services, Cloudflare, etc.): me

We are a small company. :slight_smile:

1 Like
9

That's a longer story, but I try to make it short :slight_smile:

"Time to Market" was critical as a small company with just me as a developer. So I chose the technics I was most familiar with. Volto was just too big of a beast to get into, and I did not feel comfortable tackling this at the same time, while delivering a quality product.

I'm also not sure if it makes sense, to have a Frontend, which is capable of serving the Editors and the public at the same time for a website. I see that use case for an intranet. But for a public website, it made sense to me to use the Plone Classic frontend because I was able to optimize it for user input and did not have to care about Anonymous users. The same is true if I had used Volto as frontend for the backend, but there was the knowledge constraint, and it looked just too complex at the time.

My goal is to be close enough to Plone standard to go to Volto, or build a dedicated Frontend optimized for User Input, in the future.

Regarding Blocks and layout. My implementation is very similar, the most significant difference is, that blocks on my end are regular DX types. (Not sure if this statement is still accurate)

10

@maethu so just out of interest. If the hydra PoC was finished and working 100%, would you have likely used it instead?

Essentially what it does is let you use the volto blocks model and most of the volto editor and admin UI, but it makes it easy to integrate into any frontend. So the end result is something that looks and feels like volto but lets you directly DND blocks and edit their text, but on your own frontend and without having to know how volto works

It is exactly your case that I had in mind when kicking off hydra. Because we are also a small company. Hydra is how we built https://digitalarchives.landregistry.gov.uk/ (a case study I will also post with similar benefits to what you have seen)

1 Like
11

Thanks for pointing this out. It does look interesting! Did not know it exists!

12

Interesting project, thanks for sharing.

  • Plone Classic MapWidget based on leaflet.js and OpenStreetMap. Stores GEOJson, which can be easily indexed in Elasticsearch. Support for plone.supermodel (Import/Export). Single/Multi Value. Points/Polygons/etc.

I would be interested in learning a bit more about this one. Is there any UI for editing shapes, or you have to do it somewhere else and export the geojson? Do you have maps that are aggregated from multiple content items?

  • Generic Widget based on a JSONField’s schema (Only supports a minimal number of types so far.
  • Support importing JSONField with schema for plone.supermodel

These would be nice to have in Plone core.

  • Our own WorkingCopy/Staging Implementation.

How is it different from plone.app.iterate in terms of user experience?

  • RestAPI Performance improvements.

Anything we should consider adding to plone.restapi, or are the improvements tied to project-specific assumptions?

cheers,
David

13

Impressive project. Thanks for sharing.

Regarding Blocks and layout. My implementation is very similar, the most significant difference is, that blocks on my end are regular DX types. (Not sure if this statement is still accurate)

At iMio (https://www.imio.be), we implemented a "Dexterity block" approach a few years ago. We'd be interested to see what your code looks like.

We're also interested in your z3c.form generic JSON field widget.

Thanks again for sharing.

14

@djay The solution is really fascinating. I do see the advantage regarding upgrades.

But I'm heading in a different direction.

My goal is to have a backend optimized to enter data. I don't want to marry the edit UI into the frontend.

Here is an example of the edit experience in our Backend:

Advantages:

And yes you need a somewhat big screen.

1 Like
15

@djay I did a quick screen recording of how the backend/frontend works.
I hope that helps to understand what the goal is.

2025-02-17 15.14.00

16

Since I'm doing screen capturing righ now... I made another one for the mapwidget integration :slight_smile:

  • multiple shapes per map
  • single shap per map
  • It's possible to use a JSONField with the mapwidget, or the dedicated geojsonfield.
  • default value (default coordinates, zoom level) per field.
    2025-02-17 15.38.59

Yes it does support a variation of shapes. But only the ones geojson also supports, like lines, points, polygons and rectangles.

No, there is no aggregated view so far. On my end the frontend does that. But this should not be hart to implement.

--> Currently it supports Date/Time, Boolean, text and radio inputs based on a JSONScheam and it is implemented using VueJS. What are the minimal requirements to get it into the core?

The experience is similar, with some key differences:

  • The policy is more or less hard coded. Since Blocks are DX types as well on my end it is important to only stage the current container and its blocks not the whole structure or only the container. This means there is a optimized copy mechanism, which copies the content, updates _tree, updates Blocks, updates references, etc. And the same in reverse, once the working copy gets applied to the original. I use the same feature to create versions of a content.

Mostly very opinionated changes and preventing slow queries if you combine fullobjects=1 and include_items=1, expand navigation, etc. I cannot really control the frontend developers from making those mistakes, So I changed the default behavior.

1 Like
17

The code for the DX Blocks is here GitHub - webcloud7/wcs.simplelayout (Just no official release/documentation, because I'm the only one using it).

Recarding JSONField (schema) widget
@davisagli @sverbois I'm on the plone discord server, reachable via @Mathias I'm happy to show what is there and discuss the topic.

18

Thanks for the info.

Currently [the JSONField widget] supports Date/Time, Boolean, text and radio inputs based on a JSONSchema and it is implemented using VueJS. What are the minimal requirements to get it into the core?

I guess this would involve implementing the widget in patternslib and/or plone/mockup, then adding a z3c.form widget for the field which enables that pattern. I'm not sure if it's acceptable to add something built with Vue or not. Mostly I am working on Volto and plone.restapi these days...

Support importing JSONField with schema for plone.supermodel

At least this part should be pretty straightforward. It should go in plone.schema where the field is.

19

@maethu I don't think this project using hydra would be much different. The right sidebar is your 'data" view of the content, either at the content item level, or inside block settings. This is just volto instead of classic.

image
image2874Ă—1572 416 KB

image
image2878Ă—1570 373 KB

The big difference is that to customise that sidebar you do need to use voltos framework for doing so.. except if you just stick to content type schemas and block schemas that is only simple config, no code. We intend to make this configurable without having to rebuild volto soon.
Of course for your purpose its easier for you to stick to plone classic which you obviously know better.
You would have to do less work for some complex blocks because the containment UI would be given for free.
The other big difference is real-time updates. With hydra changes to the frontend preview happen in realtime rather than after you save since they don't have to go back to the server to be received by the frontend.