How to write an Ubuntu Upstart job for Celery (django-celery) in a virtualenv

Question

I really enjoy using upstart. I currently have upstart jobs to run different gunicorn instances in a number of virtualenvs. However, the 2-3 examples I found for Celery upstart scripts on the interwebs don't work for me.

So, with the following variables, how would I write an Upstart job to run django-celery in a virtualenv.

Path to Django Project:

/srv/projects/django_project

Path to this project's virtualenv:

/srv/environments/django_project

Path to celery settings is the Django project settings file (django-celery):

/srv/projects/django_project/settings.py

Path to the log file for this Celery instance:

/srv/logs/celery.log

For this virtual env, the user:

iamtheuser

and the group:

www-data

I want to run the Celery Daemon with celerybeat, so, the command I want to pass to the django-admin.py (or manage.py) is:

python manage.py celeryd -B

It'll be even better if the script starts after the gunicorn job starts, and stops when the gunicorn job stops. Let's say the file for that is:

/etc/init/gunicorn.conf

Answer 1

You may need to add some more configuration, but this is an upstart script I wrote for starting django-celery as a particular user in a virtualenv:

start on started mysql
stop on stopping mysql

exec su -s /bin/sh -c 'exec "$0" "$@"' user -- /home/user/project/venv/bin/python /home/user/project/django_project/manage.py celeryd

respawn

It works great for me.

I know that it looks ugly, but it appears to be the current 'proper' technique for running upstart jobs as unprivileged users, based on this superuser answer.

I thought that I would have had to do more to get it to work inside of the virtualenv, but calling the python binary inside the virtualenv is all it takes.

Answer 2

Here is my working config using the newer systemd running on Ubuntu 16.04 LTS. Celery is in a virtualenv. App is a Python/Flask.

Systemd file: /etc/systemd/system/celery.service

You'll want to change the user and paths.

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=nick
Group=nick
EnvironmentFile=-/home/nick/myapp/server_configs/celery_env.conf
WorkingDirectory=/home/nick/myapp
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target 

Environment file (referenced above):/home/nick/myapp/server_configs/celery_env.conf

# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/home/nick/myapp/venv/bin/celery"

# App instance to use
CELERY_APP="myapp.tasks"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

To automatically create the log and run folder with the correct permissions for your user, create a file in /usr/lib/tmpfiles.d. I was having trouble with the /var/run/celery folder being deleted on rebooting and then celery could not start properly.

My /usr/lib/tmpfiles.d/celery.conf file:

d /var/log/celery 2775 nick nick -
d /var/run/celery 2775 nick nick -

To enable: sudo systemctl enable celery.service

Now you'll need to reboot your system for the /var/log/celery and /var/run/celery folders to be created. You can check to see if celery started after rebooting by checking the logs in /var/log/celery.

To restart celery: sudo systemctl restart celery.service Debugging: tail -f /var/log/syslog and try restarting celery to see what the error is. It could be related to the backend or other things.

Hope this helps!