About bringing websocket support to Plone

At Plone Open Garden, a few week ago, I experimented on rebasing ZServer on top of Twisted reactor and its HTTP server (instead of asyncore and medusa). Not only did that bring Python 3 support for ZServer, but also option to benefit from the whole Twisted ecosystem.

One particularly interesting possible feature made possible by Twisted is Autobahn based websocket support to Plone.

The question is: What would you do with that?

At PLOG, I made a quick demo on how to use websocket with ZeroMQ to provide instant one to many notifications from zope.events trigged by one request to all subscribed clients.

My own first real use-case is mostly just that: to allow a service to subscribe changes from a Plone site using websocket (instead of my current solution of Plone calling a webhook of that service).

It's good to acknowledge that the websocket support I'm planning, would not magically make Plone asynchronous or faster. Briefly

  • websocket would work in parallel to HTTP (inside the same server) so that any path available for HTTP could also be connected with websocket (which is detected by connection upgrade-header)

  • each connection would have its path context and session cookie saved on its state

  • any call to Plone through websocket would create normal ZPublisher request to Plone (with the session cookie for authentication), although the request could carry a websocket only marker interface to match dedicated views; response body from request would be written to websocket

  • ZMQ pub-sub requires use of custom transaction data manager so that messages are published only at the second phase of successful commit

  • ZMQ pub-sub routing is based on a single bytestring, so making complex use cases secure require some creativity (and probably extra ZPublisher calls to ensure that each subscriber is still allowed to receive the latest message)

2 Likes

One thing that comes to mind would be notifying comments to logged-in users, for example in company intranets where there are hundreds of users that can comment on each other's items.

2 Likes

Thanks. That would be a good example, because it requires targeting messages to specific users. (And, of course, should not even be too hard. The hardest thing will be to design the websocket support so that features like these could be implemented in add-ons.)

How does load balancing work on that stack?

AFAIK websocket connections are TCP connections and cannot be "load balanced" beyond balancing the initial HTTP hand shake so that the eventual connections are evenly shared between instances.

ZMQ supports unix domain sockets for pub-sub so it can distribute messages along all the instances on the same (Linux) server without external broker so that all websocket connections can be reached regardless of their instance as long as they are hosted on the same server. (To distribute messages across separate servers I would use a broker based protocol. e.g. AMQP, but I will start with the most usual use-case of multiple instances on the same server.)

For gatsby-source-plone I needed features that seem to make also discussion notifications possible. //cc @fredvd

I included that into https://hub.docker.com/r/datakurre/gatsby-source-plone/tags (datakurre/gatsby-source-plone:2019-06-21) when add-on "collective.wsevents" is activated.

2 Likes

@fredvd the latest docker image implements "new commend added" -notification with customizable websocket content-rule action:

docker run --rm -ti -p 8080:8080 datakurre/gatsby-source-plone:2019-06-23

Cool! I’ll stop now making up more use case before I have tried it myself. :stuck_out_tongue_closed_eyes:

I could try myself later to hook up the messaging to a more generic message sending action to individual users or later groups like the mail to role/group plugin which provides templates. Worked a bit on an extended mail plugin many years ago :zipper_mouth_face:

State of things update.

I haven't been able to work this the Beethoven sprint in June, but hope to continue working around the Plone Conference in October.

The current branches required on top of a Plone 5.2 buildout:

A few things missing:

  • Websocket API for calling ZServer HTTP endpoints (current implementation covers only subscribing to events; I know how to do this, but still needs time)

  • Some configuration options for old ZServer still has no effect on Twisted implementation (most important one being fast-listen=off).

  • HTTP/2-configuration option for ZServer (I've test driven implementation, but not committed that with configuration options)

  • Everything related to release: Removing all medusa code from ZServer, because it is replaced with Twisted also on Python 2 (yes, all this websocket stuff actually works on Python 2 too), fixing tests, documenting new configuration options.

I have not proposed nor planned any talks about this in particular for Plone Conference, but I'd love to host a small workshop about the possible use-cases to support and how to eventually bring this into core.

4 Likes

I'd love to see a talk from you about this topic! (just my 2c) :slight_smile:

I will visit the topic in my GatsbyJS talk on relevant part, but otherwise I was not comfortable for complete websocket talk when its other use cases and road to core is still unclear.

Oh well, I realised that I would need about 20 minutes to talk about the internals of my ZServer anyway, so I submitted a technical talk about these matters.

3 Likes

good boy :wink:

Plone Foundation Code of Conduct