Moonshot for migrating ZServer to Python 3 at PLOG 2019

I'm sorry for being off the loop for a while. Is it still that nobody has tried migrating ZServer to Python 3 yet? Or has tried and failed? @pbauer?

I'm attending Plone Open Garden 2019 and if this is still not done, I would like to try. I've used to very tightly coupled asynchronous integrations for Plone, built on top of asyncore based ZServer and would like to keep them working on Python 3, in addition to have option for using websockets. If I get PoC done at PLOG, I should be able to get more time for it during the summer.

Or alternatively someone has to convince me that WSGI is superior to ZServer and that it’s better to have async integrations with WSGI, ASGI, or something instead.

My initial plan is to replace asyncore+medusa with Twisted on Python 2 and then port the remaining parts of ZServer to Python 3. Why Twisted? Because of its maturity, ecosystem (e.g. Autobahn and XMPP), and from ZServer version history it looks like there has been time when ZServer has supported it already in the past.

I try my best to be open for feedback and ideas :slight_smile:

2 Likes

Yay!

Threadpools for the win.

I have a couple of very primitive PRs:

1 Like

So far there were no plans to port ZServer to Python 3. I'm neither the most async guy around so I cannot really comment on async-capabilities in a WSGI-setup.

My concern is currently the best-practice of deploying Plone and waitress seems fine and I also like the fact that there are options like gunicorn and uwsgi that we can use. ZServer's job as a http-Server seems to have been taken over by something well maintained and flexible.

Do you plan to use ZServer as a http-Server for the whole of Plone or are you only interested in the asyc features? Do you think it would be generally better for us to use ZServer over one of the existing WSGI-Servers?

You should probably talk to @tschorr since he tested and wrote most of the wsgi-setup in plone.reciple.zope2instance.

Thanks. It’s important to know that there is no overlapping work on-going.

My first plan is to use Twisted’s HTTP (with HTTP2 support) server as the actual http server and eg. Autobahn as websocket server (all in the same Twisted async loop - as much compatible with asyncio as possible).

(After this I need to add that this would not bring asyncio support to actual Plone requests. Only similar support on the top level as now with asyncore.)

As long as you deal with uploads and downloads of files and do not severely rearchitect that out of the publisher to happen with other middleware, yes.

Downloads are something one can (and in anything public facing, probably should) tackle elegantly through the sendfile route, uploads do remain an issue.

1 Like

Interesting text from the ancient scrolls:
https://twistedmatrix.com/pipermail/twisted-python/2001-October/000480.html

Well, it's also interesting that Zope3 did replace ZServer with Twisted already in 2006:

And it was also supported by Zope4 until removed in 2009:

I assume that after the sprint, I know, why was that, or manage to restore the support again with a Python 3 and asyncio compatible Twisted server

@datakurre could you elaborate on your use case? More specifically I'd like to better understand what makes ZServer superior for you to e.g. waitress (both use asyncore for request handling - waitress "vendors" the module now because it's deprecated and will be removed from the stdlib eventually). And what prevents you from simply using Twisteds WSGI capabilities (https://twistedmatrix.com/documents/current/web/howto/web-in-60/wsgi.html)?

The most active ones are https://pypi.org/project/collective.zamqp/, https://pypi.org/project/collective.taskqueue/, https://pypi.org/project/collective.futures/. I guess, many of these use cases could be handled with a WSGI middleware, a custom HTTP server dispatching Zope requests through WSGI or with a completely separate server calling Zope through some HTTP API. Any solution would require rewrites from me. Yet, a solution based on HTTP server with async loop would probably require less redesigning. That said, I don't miss asyncore in particular, but would like to research the possibilities of asyncio and/or Twisted.

I will look into if I could get my use cases work through WSGI interface. Although, it's been fun that with ZServer it's been possible to have custom requests classes and have add-ons registering views with layers matching only those requests and being never available for HTTP requests.

After a day at PLOG it seems that at least this would be possible, and this also was the way legacy zope.app.twisted did its twisted support.

curl --http2 -k -i https://localhost:8080
HTTP/2 200 
server: TwistedWeb/19.2.0
date: Sun, 14 Apr 2019 05:15:55 GMT
x-powered-by: Zope (www.zope.org), Python (www.python.org)
content-type: text/html; charset=utf-8
content-length: 8149
x-frame-options: SAMEORIGIN

I do have some code depending on ZServer stream iterators, so I'll still look into the non-wsgi-publisher...

2 Likes

@tschorr Do you know in detail, how we stream blobs with Waitress?

The old ZServer let worker threads start processing new requests as soon as blob file handle was returned from ZPublisher. Eventually the blobs were streamed from the main thread.

Once I noticed that Twisted WSGIService surprisingly blocks the worker thread until it has streamed the response blob (it does not use Twisted's streaming producers), I looked into Waitress, and for me it seems that it does the same.

If I benchmark a single worker thread waitress Plone against a single worker thread ZServer Plone, the old ZServer is twice as fast in serving large files, because it streams them concurrently from its main thread concurrently...

...or maybe I am missing something?

Python 2.7, Plone 5.2rc1, ZServer:

➜ ab -n 10 -c 10 http://localhost:8080/Plone/robotlab-for-macosx.zip

Benchmarking localhost (be patient).....done


Server Software:        Zope/(4.0b10,
Server Hostname:        localhost
Server Port:            8080

Document Path:          /Plone/robotlab-for-macosx.zip
Document Length:        426112439 bytes

Concurrency Level:      10
Time taken for tests:   4.443 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      4261126590 bytes
HTML transferred:       4261124390 bytes
Requests per second:    2.25 [#/sec] (mean)
Time per request:       4442.693 [ms] (mean)
Time per request:       444.269 [ms] (mean, across all concurrent requests)
Transfer rate:          936651.81 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:  4426 4429   2.4   4429    4435
Waiting:       12   39  18.1     41      63
Total:       4427 4429   2.3   4429    4435

Percentage of the requests served within a certain time (ms)
  50%   4429
  66%   4430
  75%   4430
  80%   4430
  90%   4435
  95%   4435
  98%   4435
  99%   4435
 100%   4435 (longest request)

~ took 5s 

Python 3.7, Plone 5.2rc1, Waitress:

➜  ab -n 10 -c 10 http://localhost:8080/Plone/robotlab-for-macosx.zip

Benchmarking localhost (be patient).....done


Server Software:        waitress
Server Hostname:        localhost
Server Port:            8080

Document Path:          /Plone/robotlab-for-macosx.zip
Document Length:        426112439 bytes

Concurrency Level:      10
Time taken for tests:   11.375 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      4261126940 bytes
HTML transferred:       4261124390 bytes
Requests per second:    0.88 [#/sec] (mean)
Time per request:       11375.196 [ms] (mean)
Time per request:       1137.520 [ms] (mean, across all concurrent requests)
Transfer rate:          365818.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       1
Processing:  1202 6880 3198.9   7763   11061
Waiting:      313 5775 3481.2   6942   10218
Total:       1202 6881 3198.9   7763   11061

Percentage of the requests served within a certain time (ms)
  50%   7763
  66%   8544
  75%   9396
  80%  10218
  90%  11061
  95%  11061
  98%  11061
  99%  11061
 100%  11061 (longest request)

~ took 11s 

Waiting: -rows are telling.

1 Like

Does using Twisted imply that we should be able to SSH into a Plone instance? :thinking:

2 Likes

Yes.

I tried this out now and this allows, for example, to SSH into an instant Python debug console of a single instance "ZEOless" Plone server.

1 Like

Whoa that's cool! I'm gonna have some fun with that :smiley:

And, of course, web sockets:

6 Likes

That's really nice!

:exploding_head:

I'm leaving Sorrento today, so time for a brief sprint report.

As implied by the title, my goal was not to complete ZServer Python 3 port at PLOG, but try out if I can do that and explore the possibilities allowed by the port. For that I can say that the sprint was success and I continue to complete the Python 3 port after the sprint and expect something stable around Beethoven Sprint 2019 in June (where I will migrate collective.taskqueue to support the new ZServer). If I continue to progress without issues, I hope this to be ready for Plone 6.

So, Plone 5.2 will probably ship with only WSGI option for Python 3, configuring waitress by default. But as shown above, the waitress setup may have issues when run with just a few threads on a server with a lot of big blobs (each blob download will blog a waitress worker thread until completed). But if that becomes an issue, there are other WSGI servers to try out, of course.

At the sprint I was able to:

  • replace ZServer HTTPServer with Twisted HTTP server
  • reimplement asynchronous blob streaming with Twisted FileSender
  • move ZServer WebDAV (and WebDAV source server) on top of Twisted HTTP server
  • use the same ZPublisher publish_module for HTTP as used for WSGI
  • configure Twisted to use asyncio eventloop, especially uvloop, when available
  • try out Twisted SSL support for HTTP and HTTP/2 support (based on hyper-h2)
  • try out Twisted SSH and REPL features to "SSH into Python interpreter of live Plone instance"
  • try out Autobahn with Twisted HTTP server so that WebSocket has endpoint eg. at /ws or /Plone/@@ws
  • try out Publish-Subscribe-pattern based notifications from zope.events to browsers using WebSockets and ZMQ (on Linux, with its unix domain socket support, ZMQ can be used to broadcast events across all Plone instances on the same server)

Of course, all this, with Plone 5.2rc1 on Python 3.

My changes live at my fork of ZServer at GitHub, and I'll write a more detailed blog post when it is ready to be tried out by others (currently it is a bit of a mess after all the experiments above).

My experience of Twisted was surprisingly good during the sprint. Everything just worked, documentation was good, examples were plenty and also ecosystem support looked good (ZMQ, Redis, AMQP, etc..). Therefore I am quite confident that once I get to launch our first Python 3 site later this year, I will keep using ZServer to run Plone :slight_smile:

11 Likes

/me shakes head in wonder

Thx Asko!!

1 Like

Plone Foundation Code of Conduct