I'm sorry for being off the loop for a while. Is it still that nobody has tried migrating ZServer to Python 3 yet? Or has tried and failed? @pbauer?
I'm attending Plone Open Garden 2019 and if this is still not done, I would like to try. I've used to very tightly coupled asynchronous integrations for Plone, built on top of asyncore based ZServer and would like to keep them working on Python 3, in addition to have option for using websockets. If I get PoC done at PLOG, I should be able to get more time for it during the summer.
Or alternatively someone has to convince me that WSGI is superior to ZServer and that it’s better to have async integrations with WSGI, ASGI, or something instead.
My initial plan is to replace asyncore+medusa with Twisted on Python 2 and then port the remaining parts of ZServer to Python 3. Why Twisted? Because of its maturity, ecosystem (e.g. Autobahn and XMPP), and from ZServer version history it looks like there has been time when ZServer has supported it already in the past.
So far there were no plans to port ZServer to Python 3. I'm neither the most async guy around so I cannot really comment on async-capabilities in a WSGI-setup.
My concern is currently the best-practice of deploying Plone and waitress seems fine and I also like the fact that there are options like gunicorn and uwsgi that we can use. ZServer's job as a http-Server seems to have been taken over by something well maintained and flexible.
Do you plan to use ZServer as a http-Server for the whole of Plone or are you only interested in the asyc features? Do you think it would be generally better for us to use ZServer over one of the existing WSGI-Servers?
You should probably talk to @tschorr since he tested and wrote most of the wsgi-setup in plone.reciple.zope2instance.
Thanks. It’s important to know that there is no overlapping work on-going.
My first plan is to use Twisted’s HTTP (with HTTP2 support) server as the actual http server and eg. Autobahn as websocket server (all in the same Twisted async loop - as much compatible with asyncio as possible).
(After this I need to add that this would not bring asyncio support to actual Plone requests. Only similar support on the top level as now with asyncore.)
As long as you deal with uploads and downloads of files and do not severely rearchitect that out of the publisher to happen with other middleware, yes.
Downloads are something one can (and in anything public facing, probably should) tackle elegantly through the sendfile route, uploads do remain an issue.
@datakurre could you elaborate on your use case? More specifically I'd like to better understand what makes ZServer superior for you to e.g. waitress (both use asyncore for request handling - waitress "vendors" the module now because it's deprecated and will be removed from the stdlib eventually). And what prevents you from simply using Twisteds WSGI capabilities (https://twistedmatrix.com/documents/current/web/howto/web-in-60/wsgi.html)?
The most active ones are collective.zamqp · PyPI, collective.taskqueue · PyPI, collective.futures · PyPI. I guess, many of these use cases could be handled with a WSGI middleware, a custom HTTP server dispatching Zope requests through WSGI or with a completely separate server calling Zope through some HTTP API. Any solution would require rewrites from me. Yet, a solution based on HTTP server with async loop would probably require less redesigning. That said, I don't miss asyncore in particular, but would like to research the possibilities of asyncio and/or Twisted.
I will look into if I could get my use cases work through WSGI interface. Although, it's been fun that with ZServer it's been possible to have custom requests classes and have add-ons registering views with layers matching only those requests and being never available for HTTP requests.
@tschorr Do you know in detail, how we stream blobs with Waitress?
The old ZServer let worker threads start processing new requests as soon as blob file handle was returned from ZPublisher. Eventually the blobs were streamed from the main thread.
Once I noticed that Twisted WSGIService surprisingly blocks the worker thread until it has streamed the response blob (it does not use Twisted's streaming producers), I looked into Waitress, and for me it seems that it does the same.
If I benchmark a single worker thread waitress Plone against a single worker thread ZServer Plone, the old ZServer is twice as fast in serving large files, because it streams them concurrently from its main thread concurrently...
...or maybe I am missing something?
Python 2.7, Plone 5.2rc1, ZServer:
➜ ab -n 10 -c 10 http://localhost:8080/Plone/robotlab-for-macosx.zip
Benchmarking localhost (be patient).....done
Server Software: Zope/(4.0b10,
Server Hostname: localhost
Server Port: 8080
Document Path: /Plone/robotlab-for-macosx.zip
Document Length: 426112439 bytes
Concurrency Level: 10
Time taken for tests: 4.443 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 4261126590 bytes
HTML transferred: 4261124390 bytes
Requests per second: 2.25 [#/sec] (mean)
Time per request: 4442.693 [ms] (mean)
Time per request: 444.269 [ms] (mean, across all concurrent requests)
Transfer rate: 936651.81 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 4426 4429 2.4 4429 4435
Waiting: 12 39 18.1 41 63
Total: 4427 4429 2.3 4429 4435
Percentage of the requests served within a certain time (ms)
50% 4429
66% 4430
75% 4430
80% 4430
90% 4435
95% 4435
98% 4435
99% 4435
100% 4435 (longest request)
~ took 5s
Python 3.7, Plone 5.2rc1, Waitress:
➜ ab -n 10 -c 10 http://localhost:8080/Plone/robotlab-for-macosx.zip
Benchmarking localhost (be patient).....done
Server Software: waitress
Server Hostname: localhost
Server Port: 8080
Document Path: /Plone/robotlab-for-macosx.zip
Document Length: 426112439 bytes
Concurrency Level: 10
Time taken for tests: 11.375 seconds
Complete requests: 10
Failed requests: 0
Total transferred: 4261126940 bytes
HTML transferred: 4261124390 bytes
Requests per second: 0.88 [#/sec] (mean)
Time per request: 11375.196 [ms] (mean)
Time per request: 1137.520 [ms] (mean, across all concurrent requests)
Transfer rate: 365818.47 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.2 0 1
Processing: 1202 6880 3198.9 7763 11061
Waiting: 313 5775 3481.2 6942 10218
Total: 1202 6881 3198.9 7763 11061
Percentage of the requests served within a certain time (ms)
50% 7763
66% 8544
75% 9396
80% 10218
90% 11061
95% 11061
98% 11061
99% 11061
100% 11061 (longest request)
~ took 11s
I'm leaving Sorrento today, so time for a brief sprint report.
As implied by the title, my goal was not to complete ZServer Python 3 port at PLOG, but try out if I can do that and explore the possibilities allowed by the port. For that I can say that the sprint was success and I continue to complete the Python 3 port after the sprint and expect something stable around Beethoven Sprint 2019 in June (where I will migrate collective.taskqueue to support the new ZServer). If I continue to progress without issues, I hope this to be ready for Plone 6.
So, Plone 5.2 will probably ship with only WSGI option for Python 3, configuring waitress by default. But as shown above, the waitress setup may have issues when run with just a few threads on a server with a lot of big blobs (each blob download will blog a waitress worker thread until completed). But if that becomes an issue, there are other WSGI servers to try out, of course.
At the sprint I was able to:
replace ZServer HTTPServer with Twisted HTTP server
reimplement asynchronous blob streaming with Twisted FileSender
move ZServer WebDAV (and WebDAV source server) on top of Twisted HTTP server
use the same ZPublisher publish_module for HTTP as used for WSGI
configure Twisted to use asyncio eventloop, especially uvloop, when available
try out Twisted SSL support for HTTP and HTTP/2 support (based on hyper-h2)
try out Twisted SSH and REPL features to "SSH into Python interpreter of live Plone instance"
try out Autobahn with Twisted HTTP server so that WebSocket has endpoint eg. at /ws or /Plone/@@ws
try out Publish-Subscribe-pattern based notifications from zope.events to browsers using WebSockets and ZMQ (on Linux, with its unix domain socket support, ZMQ can be used to broadcast events across all Plone instances on the same server)
Of course, all this, with Plone 5.2rc1 on Python 3.
My changes live at my fork of ZServer at GitHub, and I'll write a more detailed blog post when it is ready to be tried out by others (currently it is a bit of a mess after all the experiments above).
My experience of Twisted was surprisingly good during the sprint. Everything just worked, documentation was good, examples were plenty and also ecosystem support looked good (ZMQ, Redis, AMQP, etc..). Therefore I am quite confident that once I get to launch our first Python 3 site later this year, I will keep using ZServer to run Plone
