Plone 5.2 WSGI deployment

We (at University of Bologna) are working on our first project using Plone 5.2 / Python 3.

The main question is about production deployment. Is there someone with a medium/large installation in the wild that want to share their experience?

Our main questions:

Currently we have the current battle tested deployment: nginx + varnish + haproxy + multiple ZServer processes (ZEO client or RelStorage client) over multiple virtual server, for scaling and fail over.

With Plone 5.2/python3 there is WSGI, currently mandatory.

Any experience with different WSGI server in production? Waitress is enough for production or is it better to choose other options: gunicorn, uwsgi, ... ? pro/cons ?

What about blob file streaming? Is wsgi.file_wrapper the right solution here (https://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling)?
It is performant enough in waitress (https://docs.pylonsproject.org/projects/waitress/en/stable/filewrapper.html) or if needing performance for many/large blob files is better to choose a different wsgi server?

What about balancing?
Better following well-beaten path: multiple wsgi server http and haproxy or varnish http balancer? Or is someone experienced with specific wsgi balancer, automatic scaling, cleaver balancing, like circus or zc.resumelb or others?

What about the small gems that we had before in Plone but really closed to ZServer implementation. Like inspect running threads (kill -USR1 instance.pid) or collective.taskqueue, haufe.requestmonitoring, ... there are ready to work working alternatives?

Asko what about your work in progress on the new python 3 ZServer over Twisted?

Last but non least, what about documentation: https://docs.plone.org/manage/deploying/wsgi.html seems to me too old.

References / follow up of:

2 Likes

You are right, the documentation regarding deployment and performance of Plone 5.2 with WSGI is currently non-existent. We also lack tools like you mentioned (Sentry-integration!). The reason is simple: It is still new and only few people are using it in production.

At the conference there will be a training covering some of these topics (https://2019.ploneconf.org/training/waitress-theres-a-plone-in-my-wsgi-can-i-talk-to-the-supervisor/) by @tschorr. The trainign will yield some documentation to get you started but the bulk of the documentation and tools will only exists when we all share what we learn and write documentation about that.

Missing documentation regarding WSGI deployment is one thing.
But as noticed there are various missing things missing in the deployment pipeline: Sentry integration already mentioned, configurable mail logger integration etc.

Perhaps it would make sense to document the missing parts that require porting work or perhaps a reimplementation. Perhaps there is a chance to pool some money in order to sponsor the work on a particular feature. Speaking as contractor for the University Gent: we will likely face the same issues at the time when we go into production.

Andreas

This is a part of our todo list for the deployment. I need to work at least on the first three issues.

13-08_48_28

I've found other useful hints about wsgi deployment and a great wsgi server comparison here:
https://zope.readthedocs.io/en/latest/wsgi.html (thanks to @dataflake).

I will continue this week with bringing back support for a mail logger under WSGI.

I need haufe.requestmonitoring myself, so I've created a Pull Request. The package uses the IPubStart/IPubSuccess/IPubFailure events, so there's nothing WSGI specific here. But some fixes for Python 3 and Zope 4 compatibility were necessary.

Plone Foundation Code of Conduct