问题
I use celery to do some IO tasks, such as grab remote image, sending email to users. But celery sometimes blocked with no logs. At this time, it won't do any task i send. I have to restart it, it begin to work where it blocked.
It puzzles me for a very long time. What can i do ? And what is the best practice for distributing IO tasks with celery?
回答1:
By default, celery worker fork several processes waiting for tasks request from client. For the tasks of IO pending and your system need a larger number of concurrency that handle request concurrently. Here is the command:
celery -A tasks worker --without-heartbeat -P threads --concurrency=10
If simutanelous income requests is a lot, your concurrency level have to set higher than the size of incoming reqeust burst. The system's performance may be limited by the hardware memeory size or OS's select API. You can use celery's thread/ gevent model when concurrency is large:
celery -A tasks worker --without-heartbeat -P threads --concurrency=1000
or
celery -A tasks worker --without-heartbeat -P gevent --concurrency=1000
回答2:
you can increase the celery concurrency
manage.py celeryd --concurrency=3
where concurrency == number of processors
run shell command
grep -c processor /proc/cpuinfo
to get number of processors on your machine
来源：https://stackoverflow.com/questions/12950682/django-celery-celery-blocked-doing-io-tasks