I am running Django with Celery on top of RabbitMQ as a queue to handle some data processing tasks. I am kicking off celery tasks when a user first signs up, as well as periodically to update their data. However, I'd like to of course give priority to the tasks running users who are currently online. I noticed there was a priority setting for tasks in celery, but it seems that rabbitmq does not support this. This thread http://groups.google.com/group/celery-users/browse_thread/thread/ac3b6123d63421e5/b7740def1389e87e?lnk=gst&q=priority#b7740def1389e87e suggests have two different queues, a high priority one and a low priority one, or setting a rate limit for lower priority tasks.
Does anyone have a good workaround to implement priority? Thanks in advance!
As I understand it, and I've not yet done this, Ask covers this here:
http://docs.celeryproject.org/en/master/faq.html#does-celery-support-task-priorities [stable] http://docs.celeryproject.org/en/latest/faq.html#does-celery-support-task-priorities [latest]
Apart from this, you can push urgent tasks to some queue (let's say urgent-queue) and set consumer priorities, i.e, let all consumers pick up task from urgent-queue with high priority.
https://github.com/celery/celery/issues/3098
At consumer end, you can define x-priority argument in queues to consume from. In the below example, consumer picks up tasks from celery queue with priority 0 and from hipri with priority 10.
Example:
CELERY_QUEUES = (
Queue('celery', Exchange('celery', type='direct'), routing_key='celery',
consumer_arguments={'x-priority': 0}),
Queue('hipri', Exchange('hipri', type='direct'), routing_key='hipri',
consumer_arguments={'x-priority': 10}),
)
来源：https://stackoverflow.com/questions/3379332/workaround-for-celery-task-priority-on-rabbitmq