Celery: Worker with concurrency and reserved tasks only running 1 task

Question

Some of the tasks in my code were taking longer and longer to execute.

Upon inspection I noticed that although I have my worker node set to concurrency 6, and 6 processe

Answer 1

I'm not sure if it's your use case, but I ran into similar problems when I had a mix of long and short tasks. Basically what happened is that at some point a process could start a very long running task, while prefetching a few other tasks, preventing them from being consumed by other processes. So I disabled the prefetching stuff, which is useful only if you're running a lot of short tasks.

To disable the prefetch, you need Celery 3.1+ and the Ofair setting, for instance:

celery -A proj worker -l info -Ofair