问题
I need to build a system which handles two types of tasks. One type can create more tasks of itself or of the other type. There will be very few workers (2-3) and only one host. The most important requirement is that the system should handle restarts gracefully: i.e. on restart, tasks that were in progress should start from scratch and the workers should pick up tasks which were queued prior to restart. Looking at Celery it appears to be suitable for this use case. However, I have a couple of questions:
1) Is Celery able to handle restarts of the whole system as described? (Celery + workers + broker, everything restarts, complete power failure).
2) What is the best broker to use for this scenario? I would prefer SQLAlchemy + SQLite as it is more "self contained", but if RabbitMQ is the right choice, given the requirement to handle full restarts gracefully, then I can go with it.
回答1:
1) Is Celery able to handle restarts of the whole system as described? (Celery + workers + broker, everything restarts, complete power failure).
Yes, When you create your own queue on RabbitMQ ( Persistent queue ) even after the restart of your server the task will be resumed and will be carried out by Celery.
2) What is the best broker to use for this scenario? I would prefer SQLAlchemy + SQLite as it is more "self contained", but if RabbitMQ is the right choice, given the requirement to handle full restarts gracefully, then I can go with it.
USe RabbitMQ. we are using RabbitMQ + Celery in production. So i would suggest you to use RabbitMQ as a broker.
Celery + RabbitMQ Best Practice
Message Reliability
Notes :
- Use more Queues (ie. not just the default one)
- Use priority workers
- Use Celery's error handling mechanisms
来源：https://stackoverflow.com/questions/29739336/can-celery-tasks-survive-restart