How to run celery as a daemon in production?

Question

i created a celeryd file in /etc/defaults/ from the code here:

https://github.com/celery/celery/blob/3.0/extra/generic-init.d/celeryd

Now when I want to run

Answer 1

I am not sure what you are doing here but these are the steps to run celery as a daemon.

1. The file that you have referred in the link https://github.com/celery/celery/blob/3.0/extra/generic-init.d/celeryd needs to be copied in your /etc/init.d folder with the name celeryd
2. Then you need to create a configuration file in the folder /etc/default with the name celeryd that is used by the above script. This configuration file basically defines certain variables and paths that are used by the above script. Here's an example configuration.
3. This link Generic init scripts explains the process and can be used for reference

Answer 2

I generally use supervisor (plus django-supervisor) for this purpose. That way, you don't need to figure out how to daemonize each process in your application (of which you have at least a webserver hosting django, plus celery, plus realistically whatever other middleware you use to support both of those). Supervisor knows how to run itself as a daemon, and all your other processes run as children of supervisor.

Answer 3

Note: in ubuntu 16.04 my anser with the .conf file is not working anymore.

I created a .service file and put this in /etc/systemd/system/

i can use

sudo service myservice status

sudo service myservice start

sudo service myservice stop

as commands

e.g. this file:

myservice.service:

[Unit] 
Description=My celery worker 

[Service]
WorkingDirectory=/srv/my-project-path
User=buildout
Group=buildout
Restart=on-failure
RestartSec=20 5
ExecStart=/srv/my-project/bin/django celeryd -BE

[Install]
WantedBy=multi-user.target
Alias=myservice.service

note i use buildout, so in setad of bin/django most users need to use the path to python and use mange.py in stead.

base upon: http://minecraft.gamepedia.com/Tutorials/Ubuntu_startup_script (see the with systemd section)

Answer 4

I found this link extremely useful: How to write an Ubuntu Upstart job for Celery (django-celery) in a virtualenv

tweaking it a bit.. I have a celery worker running using this script:

(using ubuntu upstart)

named iamcelery.conf and placed it in /etc/init (note: not init.d)

# iamcelery -runs the celery worker as my virtual env user
#
#
# This task is run on startup to start the celery worker as my vritual env user

description "runs the celery worker"
author "michel van Leeuwen <michel@iamit.nl>"

start on runlevel [2345]
stop on runlevel [!2345]

# retry if ended unexpectedly
respawn
# limit the retries to max 15 times with timeouts of 5 seconds
respawn limit 15 5

# Time to wait between sending TERM and KILL signals
kill timeout 20

task
script
  exec su -s /bin/sh -c 'exec "$0" "$@"' <place here your unprovilegd username> -- srv/<here the path of your django project>/bin/django celeryd -BE -l info
end script

now you can start this scipt (it starts on server startup as well):

sudo start iamcelery

or stop:

sudo stop iamcelery

or check its status:

sudo status iamcelery

I am not quit sure this is the neatest way.... however... after a long trial and errors trying to get the initd scripts to work.... ( without succes) ... this finally works.

Edit 8 june 2013 My script given here seemed to runs as a root in the end. Now I changed this:

script
  su <place here your unprovilegd username>
  cd /srv/<here the path of your django project>/
  exec bin/django celeryd -BE -l info
end script

into:

script
  exec su -s /bin/sh -c 'exec "$0" "$@"' <place here your unprovilegd username> -- srv/<here the path of your django project>/bin/django celeryd -BE -l info
end script

and this works, with all the credits to the answer to this question: How to write an Ubuntu Upstart job for Celery (django-celery) in a virtualenv

Edit 5 sept 2013

There is one small thing left: I have to do ctrl-c after the start command in the console (and do a status check after this one): In case somebody knows this: leave in the command, and I can update this answer...

Answer 5

As Marcin has explained in his answer that supervisor is usually what people end up using but if you are looking for something which could work with python3 and can't wait for supervisor's version 4 which I think will have the support for python3 then you can go with circus. After installing it, you just need to have a circus.ini file which will have all the processes which you want to daemonize and then run that sample circus.ini may look like:

[watcher:celery]
cmd = full_path/python3.4 full_path/manage.py celeryd -B -l info

[watcher:celerycamera]
cmd = full_path/python3.4 full_path/manage.py celery events --camera=djcelery.snapshot.Camera

[watcher:dceleryflower]
cmd = full_path/python3.4 full_path/manage.py celery flower -A your_app_name --basic_auth=username:password --port=5555 

if you want some more details I have a post related to the same here. Hope it saves someone some time. Thanks