RQ concurrency with supervisord?

时光毁灭记忆、已成空白 提交于 2019-12-23 16:19:04

问题


All, I'm attempting to 'force' RQ workers to perform concurrently using supervisord. My setup supervisord setup seems to work fine, as rq-dashboard is showing 3 workers, 3 PID's and 3 queue (one for each worker/PID). Supervisord setup is as follows (showing only worker 1 setup, 2 more workers are defined below this one):

[program:rqworker1]
command = rqworker 1 
process_name = rqworker1-%(process_num)s
numprocs = 1 
user = username 
autostart = True
stdout_logfile=/tmp/rqworker1.log
stdout_logfile_maxbytes=50MB

The issue is when I send 3 jobs concurrently, the total time to run is x3 that of a single task (namely, total time is linear with number of tasks, this scales to x4,x5, etc..). It seems no concurrency is available. I also implemented a primitive load-balancing by sending new jobs to the queue with minimum started+queued jobs, that works fine (jobs are observed to be spread evenly among queues).

Why would this setup not allow concurrency?

Any considerations regarding the setup i'm missing?

Note that rq-gevent-worker package (which worked great earlier w.r.t. concurrency/RQ) is no longer available as I migrated to PY3 and gevent itself is not yet supported on PY3. But this gives my a clue that concurrency is possible.


回答1:


Modifying my comment from above into an answer...

Using supervisord to run multiple rqworker processes in parallel is an intended pattern in python-rq, so don't be concerned that you're "forcing" it. You actually have the right idea.

On the other hand, writing your own load balancing algorithm is an anti-pattern: that's exactly what python-rq does for you.

If you want to split up work between three workers, then they should all listen to the same queue. Try removing two of your supervisor config blocks, and in the one block that remains, change numprocs to 3. If you submit three jobs rapidly to that queue, you should see three workers executing concurrently.



来源:https://stackoverflow.com/questions/29928166/rq-concurrency-with-supervisord

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!