Does the number of celeryd processes depend on the --concurrency setting?

Question

We are running Celery behind Supervisor and start it with

celeryd --events --loglevel=INFO --concurrency=2

This, however, creates a process graph that is up to three layers deep and contains up to 7 celeryd processes (Supervisor spawns one celeryd, which spawns several others, which again spawn processes). Our machine has two CPU cores.

Are all of these processes working on tasks? Are maybe some of them just worker pools? How is the --concurrency setting connected to the number of processes actually spawned?

Answer 1

You shouldn't have 7 processes if --concurrency is 2.

The actual processes started is:

• The main consumer process

  Delegates work to the worker pool

• The worker pool (this is the number that --concurrency decides)

So that is 3 processes with a concurrency of two.

In addition a very lightweight process used to clean up semaphores is started if force_execv is enabled (which it is by default i you're using some other transport than redis or rabbitmq).

NOTE that in some cases process listings also include threads. the worker may start several threads if using transports other than rabbitmq/redis, including one Mediator thread that is always started unless CELERY_DISABLE_RATE_LIMITS is enabled.