Asynchronous tasks with Plone 5.x and WSGI

The current asyncore based ZServer allows fun asynchronous features, which I have been relying a lot (collective.zamqp integrates Plone with RabbitMQ, collective.taskqueue runs simpler worker queue next to Plone without extra services and collective.futures frees Zope worker thread to server other requests while eg. waiting for a response for external service). Many may also still rely on Zope clockserver.

Obviously non of the above work with WSGI, which is fully synchronous in nature, so I should start planning replacements :slight_smile:

Normal WSGI does not support anything async, so the only pure option probably is a middleware, which queues tasks for external workers after successful transaction. If worker need ZODB access, it probably needs to call Plone using restapi and its authentication.

Using some non-standard asynchronous WSGI server would allow async operations in middle-ware, but would require choosing the server and async solution and possible locking into some specific server (eg. uwsgi or unicorn).

Finally, there’s an option to implement ZServer on top of asyncio :slight_smile:

So, what would you prefer?

Queues, and JWT authenticated external workers fetching their context and such via REST using the JWT itself as the instruction payload sounds like the thing which would cover the most use-cases.

This solves an another problem - having a dediated worker for known external dependencies (OAuth, backend based analytics or somesuch). This would also be a nice thing to have in the Plone toolbox.

This is probably the simplest solution, and should work with any WSGI server with a simple queuing middleware. Passing JWT for the user triggering the task should be enough for most use cases (to avoid managing special secrets for the workers), with some exception handling for those rare cases where JWT gets expired before the worker has execute the task

I'm not very fond of the idea of ceding full control of the user session to a 3rd party (potentially pluggable) message queue.

In any case plone.restapi needs a mechanism for issuing different kinds of application tokens for various headless clients going forwards.

plone.restapi also needs a two-tiered system where you get a longer lifetime refresh token with which you can refresh your stale tokens.

And this is how the current ZServer works. The benefit for having async worker in the same async loop with ZServer is that it could intercept responses and "hold" it until some async task have been completed. This is how collective.futures work (as long as there are unsolved futures, it will retry the request internally while browser remains waiting).

This would be most work for these options (aiohttp could replace medusa, thread executor could replace ZRendevouz, but a lot of work remains to adapt to Zope request, response, querystring demarshalling, etc...), but would allow very fancy stuff with aiohttp middlewares before and after ZPublisher.

Also, this allows async tasks to call Plone views, which are not visible for regular HTTP requests. I don't claim it's more secure than having HTTP calls to public views with proper token authentication, but it is a bit simpler for trusted code.

I agree, but this needs more thought. Anything requiring manually managet secrets would not work (eg. I wonder, how often have zope clockserver user credentials been changed in general...)

Yep, would not like to see that go with the bathwater.

Sounds like a fun sprint.

1 Like

Fun enough for a one more Sauna sprint?

1 Like

Have you considered gunicorn's asyncio workers? There's also been some work to define what wsgi could mean in async world. Sort of.

That was the second option in the first post, still under consideration, but as you said, it's "short of" and requires playing with gevent/greenlets. In my understanding async WSGI would require lock-in to a specific WSGI server (gunicorn, uwsgi, pulsar...). That makes it hard to see benefits over aiohttp based ZServer (besides a little less work).

What is current state of doing async tasks in Plone 5.2/Python 3? Has the situation become a bit more mature or are we still seeking for options?

1 Like

Same question here.
I have a project on Plone 5.1 which uses collective.taskqueue on Zserver. It looks like I'll need to change how I'm doing async before upgrading to Plone 5.2.

And I am sorry that I cannot update that to WSGI based Plone. collective.taskqueue does have a branch that works with my ZServer branch on Python 3 and Plone 5.2.x. I have been running Python 3 ZServer with Plone 5.2.1 in production about a year now, but have not been using collective.taskqueue in those projects yet (but much more heavy Camunda + microservices + Plone REST API setup instead due to their requirements). That said, @gforcada has been interested in trying out ZServer with c.taskqueue + Redis...

So start looking at Redis or abandon c.taskqueue for something that works with Plone 5.2?

c.taskqueue Python 3 branch works with my ZServer branch with or without Redis; the issue is that I have not been able to do and maintain released versions of those (not mentioning plone.recipe.zope2instance-support, which has no ZServer support on Python 3)

1 Like