Running supervisord from the host, celery from a virtualenv (Django app)

Question

I'm trying to use celery and redis queue to perform a task for my Django app. Supervisord is installed on the host via apt-get, whereas celery resides in a specific virtualenv on my system, installed via `pip.

As a result, I can't seem to get the celery command to run via supervisord. If I run it from inside the virtualenv, it works fine, outside of it, it doesn't. How do I get it to run under my current set up? Is the solution simply to install celery via apt-get, instead of inside the virtualenv? Please advise.

My celery.conf inside /etc/supervisor/conf.d is:

[program:celery]
command=/home/mhb11/.virtualenvs/myenv/local/lib/python2.7/site-packages/celery/bin/celery -A /etc/supervisor/conf.d/celery.conf -l info
directory = /home/mhb11/somefolder/myproject
environment=PATH="/home/mhb11/.virtualenvs/myenv/bin",VIRTUAL_ENV="/home/mhb11/.virtualenvs/myenv",PYTHONPATH="/home/mhb11/.virtualenvs/myenv/lib/python2.7:/home/mhb11/.virtualenvs/myenv/lib/python2.7/site-packages"
user=mhb11
numprocs=1
stdout_logfile = /etc/supervisor/logs/celery-worker.log
stderr_logfile = /etc/supervisor/logs/celery-worker.log
autostart = true
autorestart = true
startsecs=10

stopwaitsecs = 600

killasgroup = true

priority = 998

And the folder structure for my Django project is:

/home/mhb11/somefolder/myproject
├── myproject
│   ├── celery.py       # The Celery app file
│   ├── __init__.py     # The project module file (modified)
│   ├── settings.py     # Including Celery settings
│   ├── urls.py
│   └── wsgi.py
├── manage.py
├── celerybeat-schedule
└── myapp
    ├── __init__.py
    ├── models.py
    ├── tasks.py        # File containing tasks for this app
    ├── tests.py
    └── views.py

---

If I do a status check via supervisorctl, I get a FATAL error on the command I'm trying to run in celery.conf. Help!

p.s. note that user mhb11 does not have root privileges, in case it matters. Moreover, /etc/supervisor/logs/celery-worker.log is empty. And inside supervisord.log the relevant error I see is INFO spawnerr: can't find command '/home/mhb11/.virtualenvs/redditpk/local/lib/python2.7/site-packages/celery/bin/‌​celery'.

Answer 1

Path to celery binary is myenv/bin/celery whereas you are using myenv/local/lib/python2.7/site-packages/celery/bin/cel‌‌​​ery.

So if you try on your terminal the command you are passing to supervisor (command=xxx), you should get the same error.

You need to replace your command=xxx in your celery.conf with

command=/home/mhb11/.virtualenvs/myenv/bin/celery -A myproject.celery -l info

Note that I have also replaced -A parameter with celery app, instead of supervisor configuration. This celery app is relevant to your project directory set in celery.conf with

directory = /home/mhb11/somefolder/myproject

On a side note, if you are using Celery with Django, you can manage celery with Django's manage.py, no need to invoke celery directly. Like

python manage.py celery worker
python manage.py celery beat

For detail please read intro of Django Celery here.