I made a previous post about interfacing with the Plone control panel using React and the REST API. One of my targets, or shall we say 'features' that I'd like to implement is a Server -> Client connection using SocketIO. Every time an administrator creates an item I fire a subscriber event which triggers a SocketIO emit. I want the client (using the React app) to receive a notification or an automatic content update when say, a new News Item is created...
So my question is on how to implement this, I've looked into a few options but my noobness in the whole websocket area is preventing me from figuring out a solution.
The Python SocketIO doesn't really show any solutions that I can see would be relevant to Plone. As an alternative I tried to see how people have implemented SocketIO with Django but yeah..?
If I create a SocketIO server separately how can I make an emit on the Plone event. There isn't anyway to connect per-say to the server.
If I want to create a parallel instance to Plone that listens on a separate port, how would I go about that...
It's a little tricky, my words are a little short, so maybe someone understands this better and can offer some ideas on how to get this working.
Plone (IObjectCreatedEvent) -> SocketIO emit update -> Client on React app is listening, and receives update (Component code runs)
I have to say that I find it a bad idea to do a reverse connection server -> client. Everything is the world is designed to make outgoing connections from clients and incoming connections to servers easy; yet it often does not work correctly. Everything in the world is designed to make this reverse connection difficult if not impossible (think firewalls and NAT, with sometimes 2 or 3 NAT levels and several firewalls on the way including the sometimes nasty things installed on the client computer, not to forget that the browser Javascript VM is not primarily designed to be a server).
While in a very special setup (think clients on an internal and isolated network) this kind of stuff could be implemented, that's not a common configuration these days. So you will not find easily sample code and advice for this project.
If I had to something like that, I'd not even use Zope events; I'd base this part on Plone RSS implementation, if I could not just use it directly. If for some reason you can't use published stuff but need get data right after items creation, you could create a special feed that would advertise 'immediately' all items.
Advantages:
working sample code on the server (the RSS Plone implementation)
javascript samples everywhere on the net for the client side.
protocol already tested, firewall-proof (using straight http) and secured with https
no need to open ports, the access point is an URL on the server.
Thanks for enlightening me, I was thinking to basic socket IO. I'd not consider it nonetheless until really needed for lack of server sample code and having to implement a protocol.
The big hill to climb here is that ZServer is currently not really well geared for long standing connections. You'd somehow have to have a separate thread pool for managing those or otherwise the per thread caching would get you. This can be partially mitigated via configuring a non-caching (or low caching) separate ZEO client with a large thead pool just for serving the socket clients.
I think we'll be (at least indirectly) tackling hauling the under the hood stuff enabling that to work over to Zope4 in Sauna Sprint this year. More people are more than welcome to join us.
Or one can have a separate non-Plone/Zope process / service to which you push events via a message queue from Plone and have your webfront js client register for those subscription channels. AFAIR @datakurre did something like the latter route that a couple of years back.
what I don't see is how it could be possible to 'upgrade' an existing HTTP connection to work with another process. Unless it's more trivially a redirection to a new connection with another server port ?
Something like this happens regularly in a simple Web server: the main server "listen"s at a socket for connection requests; on a connection "accept", it (maybe) forks handing over the file descriptor for the accepted connection to the child. This, then, can do whatever it wants, e.g. perform a TLS (or WebSocket) upgrade.
I was not clear enough, I was thinking to a 'separate non-Plone/Zope process' as said by @Rotonen and I was understanding this as something started independently, not a child (it's not obvious to start a true independent process from Plone code and making it inherit something, I think it could be done but only with core changes)
Edit: I forgot there is another reason for a separate process: Zeo.
I meant it is a definite option to have a completely separate service for pushing events to the browsers and have Plone push events to that service via a message queue. @datakurre explained this well above and gave a set of technologies with which to patch such a thing together.
This stuff (service oriented architecture design) is hard, communication itself is hard - iteration is expected. Thank you for persisting.
I had a use case requiring realtime WAMP communications to a messaging router (Crossbar). In this case, the recipient of the messages was a Windows service running behind a firewall, but it could just as well be a React app given that there exists a solid Javascript WAMP client as well.
At the time all the Python WAMP client implementations I knew of were asyncio only, so I ended up creating a small Twisted-based proxy that keeps a connection open to the WAMP router and accepts JSON messages from Plone over a websocket connection.
On the Plone side, there's e.g. collective.websocketclient to manage websocket connections to the proxy and help with sending messages from Plone event subscribers. It maintains a persistent connection on the Plone client and reconnects if the connection drops.
Both the proxy & websocket client are more of a proof-of-concept quality than production-grade, but they are known to work.
This looks interesting, I definitely want to check out the collective.websocketclient...
I did a bit more playing around and sort of fell on creating a 3-way system using React, Plone and Flask + SocketIO.
So I have Flask wrapped in SocketIO running locally and then connect to the SocketIO server in a Plone subscriber event and then simply emit a message. The React side also connects and receives the messages and so on...
The only thing I don't like with a method like this is that 1) it completely segments any logic in my websocket events which I may want to handle in Plone itself, so it works purely as a middle-man system and 2) the whole idea of connecting to a SocketIO server on every event and then emitting a message sort of defeats the purpose of a websocket connection in my eyes. I'd sort of like to initiate the websocket connection throughout the entirety of my application up time and then purely emit messages directly to my server, though I'm not sure how to achieve this..?
Maybe initiate the websocket connection in my application __init__.py..?
This is an interesting topic in my eyes as we move more into the SPA movement... In-fact I'd be willing to write and include a workflow for creating socket events in the Plone documentation if there's enough to build a set workflow for this...
How'd you sculpt that bit togeter? Are you limited to as many parallel subscribers as you have WSGI workers for or do you have a solution in there for juggling multiple subscribers on a single WSGI worker?
I would not expect big problems using shared websocket client connections for your application -- in a way similar to the shared use of (relational) database connections. In this analogy, it may or may not be possible to use the same connection in different threads (depending on the thread safety of the connection objects. Again in this analogy, questions of transactional safety may or may not be relevant.
A bigger problem would be the implementation of websocket server connetions inside Plone itself. The Plone/Zope infrastructure is strongly request based and hides communication details. An important websocket purpose is the establishment of a connection between the browser and the server which remains open across multiple requests and especially allows the server to send messages to the browser without being asked for it explicitely in a request. This does not fit well with Plone's request oriented processing model - and it requires the exposure of communication details.
I do not see the biggest problem in ZServer. It already supports "FTP" which is a protocol based on long standing connections. It is fairly easy to extend (a long time ago, I implemented an NNTP extension for ZServer -- another protocol with long standing connections). The problem is how to handle an incoming websocket connection: its communication structure is not request oriented and therefore fits bad with Plone's basic processing model.
@dieter In this case I would imagine it's only a one-way system in which the Plone app connects, emits and closes the connection. My React app is then connected to SocketIO which emits a response to all clients, in this case the React app would receive the message.
But you're right in that receiving a websocket message in Plone would prove difficult, I guess any React -> Plone events would have to occur through the Plone REST API instead...