问题
I have 4 celery workers each with concurrency of 6.
I have users submitting varying number of jobs (from 1 to 20).
How do I ensure that each user's job get equal processing time, and that one user's job do not fill up the queue forcing other user's jobs to wait.
I am afraid if the workers are ending up going through all the jobs submitted by the first user, the other user's queued jobs must wait first user to finish, an inconvenience.
Is there a way to make the celery workers aware of one user's jobs holding up other user's queued jobs . Instead can I run maximum one job from each user at any given time?
I have one queue which I submit all the user's jobs to, would I need to make a queue for each user and somehow have round-robin strategy to pull one job from each user's queue?
回答1:
At the moment Celery doesn't support priority queues.
Making a queue for each user and scheduling them based on round-robin algorithm seems to be a lot of work.
One simple way to solve your problem is to create a temporary table & store the incoming task details. Send first received task to celery. By the time it's completed, you might have received a lot of tasks from various users. Now based on the user id, completed tasks & uncompleted tasks, you can send most appropriate task to celery for execution.
来源：https://stackoverflow.com/questions/25543085/celery-make-sure-workers-are-not-running-only-jobs-from-one-user