Celery difference between concurrency, workers and autoscaling

Question

In my /etc/defaults/celeryd config file, I\'ve set:

CELERYD_NODES=\"agent1 agent2 agent3 agent4 agent5 agent6 agent7 agent8\"
CELERYD_OPTS=\"--a         


        

Answer 1

Let's distinguish between workers and worker processes. You spawn a celery worker, this then spawns a number of processes (depending on things like --concurrency and --autoscale, the default is to spawn as many processes as cores on the machine). There is no point in running more than one worker on a particular machine unless you want to do routing.

I would suggest running only 1 worker per machine with the default number of processes. This will reduce memory usage by eliminating the duplication of data between workers.

If you still have memory issues then save the data to a store and pass only an id to the workers.