Restart celery beat and worker during Django deployment

问题

I am using celery==4.1.0 and django-celery-beat==1.1.0.

I am running gunicorn + celery + rabbitmq with Django.

This is my config for creating beat and worker

celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach

During Django deployment I am doing following:

rm -f celerybeat.pid
rm -f celeryd.pid
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach
service nginx restart
service gunicorn stop
sleep 1
service gunicorn start

I want to restart both celery beat and worker and it seems that this logic works. But I noticed that celery starts to use more and more memory during deployment and after several deployments I hit 100% memory use. I tried different server setups and it seems that it is not related.

回答1:

rabbitmq may be to blame for high memory usage. Can you safely restart rabbit?

Also can you confirm that after a restart there is the expected amount of workers?

回答2:

You are starting 2 new workers for every deployment without stopping/killing the previous workers.

During deployment, stop the existing workers with

kill -9 $PID
kill -9 `cat /var/run/myProcess.pid`

Alternatively, you can just kill all the workers with

pkill -9 celery

Now you can start workers as usual.

celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach

来源：https://stackoverflow.com/questions/48531806/restart-celery-beat-and-worker-during-django-deployment