Running multiple instances of celery on the same server

Question

I want to run two instances of celery on the same machine. One is for an 'A' version of my application, the other is for the 'B' version.

I have two instances, which I start like this:

(env1)/home/me/firstapp$ celery -A app.tasks worker --config celeryconfig
(env2)/home/me/secondapp$ celery -A app.tasks worker -n Carrot --config celeryconfig

In tasks.py in each application, I create a celery instance like this:

 celery = Celery('tasks', backend='amqp', broker='amqp://guest@127.0.0..1.5672//')
 @celery.task
 def run_a_task():
     do_stuff()

In env2's task.py, how can I specify that I want to use the second celery instance from secondapp(named Carrot), rather than the first one from firstapp? I suspect I need to change something in the constructor for celery on the first line, but I don't know what to add.

Answer 1

I solved this by using a virtual host for celery.

Once the rabbitmq server is running I issue these commands:

rabbitmqctl add_user user password
rabbitmqctl add_vhost app2
rabbitmqctl set_permissions -p app2 user ".*" ".*" ".*"

Then I start celery with:

celery -A tasks worker --broker=amqp://user:password@localhost/app2

With my task, I initialize the celery object like this:

celery = Celery('tasks', backend='amqp', broker='amqp://user:password@localhost:5672/app2

Answer 2

It looks like you're using AMQP so I would recommend solving this using different exchanges.

I recommend reading this blogposts about the AMQP structure: http://blogs.digitar.com/jjww/?s=rabbits&x=0&y=0

For Celery specific information, have a look here: http://docs.celeryproject.org/en/latest/userguide/routing.html

A short version of what you could do is give the apps a different default queue:

from kombu import Exchange, Queue

CELERY_DEFAULT_QUEUE = 'app1'
CELERY_QUEUES = (
    Queue('app1', Exchange('app1'), routing_key='app1'),
)