Anyone using or considering alternatives to zc.async / plone.app.async for simple job-queue stuff? I'm contemplating doing something on this front (for media transcoding jobs), but I wanted to see what solutions folks have been using?
My goals:
Simple job queue, queue a context object and a callable that takes that context as an argument;
Don't care about scheduling, only avoiding race-conditions;
Want to be able to run something async as either long-running process or cron job (or maybe a runscript that will do either).
Light-weight, no ZODB writes except for logging job to queue, and committing final result.
although also based on plone.app.async, it's at least worth a look to see how Plume (http://blog.plumi.org) has implemented async transcoding of media files.
In ploneintranet we ran into quite some problems with plone.app.async and have started to work on a new architecture that avoids most of the drawbacks of current async stacks, by providing a full HTTP-request based async stack, so that callbacks are fully authenticated and whathaveyou.
Unfortunately this work is currently stalled on a feature branch but we hope to pick up on that after the holiday. Please let me know if you're interested in collaborating on this!
Yes, FWIW, I want to do something with (or similar to) wildcard.media, which I am using now with a fork of zc.async (release version of zc.async, by bringing in stale uuid package, breaks things [1]).
Short comments on behalf of collective.zamqp and collective.taskqueue:
c.taskqueue is probably the only standalone async solution working without external services
c.taskqueue doesn't have any monitoring OOTB
c.taskqueue supports work distribution and quaranteed delivery with redis (yet, I've only used it without redis, because using c.zamqp for complex stuff)
c.zamqp has seen production a lot, and I'm not aware of ever losing a task
c.zamqp requires RabbitMQ, but allows direct control of its routing and message structure (opinionated routing and message were the issues, which removed celery from my options)
c.zamqp requires grok (it's that old) and there's no refactoring in progress, because "it's not broken"
c.zamqp works well with externalized services / workers, c.taskqueue expects everything be done and stored in Plone