Plone.app.async / zc.async alternatives?

Anyone using or considering alternatives to zc.async / plone.app.async for simple job-queue stuff? I'm contemplating doing something on this front (for media transcoding jobs), but I wanted to see what solutions folks have been using?

My goals:

  • Simple job queue, queue a context object and a callable that takes that context as an argument;

  • Don't care about scheduling, only avoiding race-conditions;

  • Want to be able to run something async as either long-running process or cron job (or maybe a runscript that will do either).

  • Light-weight, no ZODB writes except for logging job to queue, and committing final result.

Sean

Have you looked at how collective.documentviewer uses plone.app.async?

He's looking for alternatives to plone.app.async!

https://github.com/collective/collective.celery has a decent number of people using it. It's still new but more sane.

1 Like

although also based on plone.app.async, it's at least worth a look to see how Plume (http://blog.plumi.org) has implemented async transcoding of media files.

In ploneintranet we ran into quite some problems with plone.app.async and have started to work on a new architecture that avoids most of the drawbacks of current async stacks, by providing a full HTTP-request based async stack, so that callbacks are fully authenticated and whathaveyou.

https://github.com/ploneintranet/ploneintranet/blob/async-final/docs/development/components/async.rst

Unfortunately this work is currently stalled on a feature branch but we hope to pick up on that after the holiday. Please let me know if you're interested in collaborating on this!

1 Like

I'am very happy with collective.zamqp and use it in several projects.

1 Like

Yes, FWIW, I want to do something with (or similar to) wildcard.media, which I am using now with a fork of zc.async (release version of zc.async, by bringing in stale uuid package, breaks things [1]).

[1] http://plone.293351.n2.nabble.com/Heads-up-to-zc-async-plone-app-async-users-td7572611.html

Cool, I will take a look.

Sean

we found collective.taskqueue as a good alternative, and we will try to use it in one project.

1 Like

Short comments on behalf of collective.zamqp and collective.taskqueue:

  • c.taskqueue is probably the only standalone async solution working without external services
  • c.taskqueue doesn't have any monitoring OOTB
  • c.taskqueue supports work distribution and quaranteed delivery with redis (yet, I've only used it without redis, because using c.zamqp for complex stuff)
  • c.zamqp has seen production a lot, and I'm not aware of ever losing a task
  • c.zamqp requires RabbitMQ, but allows direct control of its routing and message structure (opinionated routing and message were the issues, which removed celery from my options)
  • c.zamqp requires grok (it's that old) and there's no refactoring in progress, because "it's not broken"
  • c.zamqp works well with externalized services / workers, c.taskqueue expects everything be done and stored in Plone
  • both rely on asyncore and won't work on wsgi
3 Likes

2 posts were split to a new topic: Problem to make async part of collective.documentviews work