Stack Update: Cloud- and PostgreSQL-native Zope/Plone

[jensens (Jens W. Klein)]

Servus Plonistas!
Quick update: the stack is finally working end-to-end - at least AFAICT.

I've been using it in an actual project and thanks to some of you and your bug reports, things have hardened quite a bit.

Here's what we have now:

• zodb-s3blobs
  a generic storage wrapper for blobs in any S3 — not PostgreSQL related and works with FileStorage, ZEO and RelStorage as well (but acts as base later)
• zodb-json-codec
  library for fast pickle to/from JSON transcoding (Rust/PyO3) — see documentation
• zodb-pgjsonb
  full ZODB Storage with JSONB pickles for PostgreSQL and S3 blob tiering — see article ZODB out of the pickle jar
• zodb-convert
  new standalone generic converter, derived from RelStorage's zodbconvert
• cookiecutter-zope-instance
  now supports pgjsonb and zodb-s3blobs storage backends — see documentation
• zodb-pgcatalog
  PostgreSQL native ZCatalog replacement — see article I Cut Down a Forest and documentation
• plone.pgthumbor
  replaces all image scaling in Plone with Thumbor image transformation service: reads blobs directly from PostgreSQL and S3 while keeping Plone view-permission security. Smart auto-focal point and automatic webp conversion. See article Sleeping without a Pillow and documentation
• plone-registryfromenviron
  populate Plone registry settings from environment variables — essential for container/K8s deployments
• cdk8s-plone
  cdk8s constructs for deploying the whole stack on Kubernetes — Plone, PostgreSQL, Thumbor, S3, all wired together

---

Another thing I am currently working on is cloud-vinyl, a Kubernetes operator for managing Vinyl Cache clusters (the FOSS HTTP cache formerly known as Varnish Cache - still exists as commercial product).

---

Feedback welcome!

[tisto]

This is very impressive Jens! Thanks for sharing!

[zopyx (Andreas Jung)]

Awesome work, Jens!

A major bottleneck in Plone has always been moving or updating large object trees due to the need to reindex all sub content recursively. Renaming or moving a tree could take minutes due to reindexing. This is problem solved with the new components?

[jensens (Jens W. Klein)]

@zopyx

A major bottleneck in Plone has always been moving or updating large object trees due to need to reindex all sub content recursively. Renaming or moving a tree could take minutes due to reindexing. This is problem solved with the new components?

This is an interesting optimization. Yes, this could be optimized with the pgjsonb and pgcatalog, but I did not address it yet.

[davisagli (David Glick)]

A while ago I worked on a draft to optimize moving content by updating the catalog path of the existing metadata record instead of doing a full reindex. The results were promising but I never had time to finish it. It should probably actually go in CMFCore instead of a monkeypatch, and with a way to configure which indexes are path-dependent. Draft: Experiment with optimizing moves by davisagli · Pull Request #3834 · plone/Products.CMFPlone · GitHub

[maethu (Mathias)]

I wrote this package 10y ago: GitHub - 4teamwork/ftw.copymovepatches · GitHub

Probably did something similar and was battle tested on a lot of Plone 4 and 5 installations.

[davisagli (David Glick)]

@maethu Now I remember, my CMFPlone branch was inspired by ftw.copymovepatches, which had not been updated for Plone 6. Thanks for sharing your work on that, and sorry I forgot to give credit.

[maethu (Mathias)]

No problem at all! It’s been a long time. It just came to my mind when I saw your post. I’m glad there was something useful in there

[jensens (Jens W. Klein)]

I implemented a move/rename with bulk SQL path update and I now need better intergation testing and so moved over to zodb-pgsjonb to create a kind of DemoStorage-like fixture for it (DemoStorage can be used with zodb-pgjsonb as long as you dont have code using SQL, like pgcatalog).

This is not finished yet, but the curious ones here may have a look already.

[maethu (Mathias)]

@jensens

(DemoStorage can be used with zodb-pgjsonb as long as you dont have code using SQL, like pgcatalog).

Mind pointing me to some code, since I did my own hackish implementation for the DemoStorage to pass through actual sql queries.

[jensens (Jens W. Klein)]

@maethu

Mind pointing me to some code,

It's in the making and even with agentic-coding a non-trivial task.

[maethu (Mathias)]

I bet, if you have not seen it: ZODB out of the pickle jar - #36 by pbauer

That’s my “working“ approach