问题
In Python, I'm trying to run 150-200 processes. I have these 150 things in an array, and I've split this array up into multiple arrays of 10 elements each.
Now, I run a Multiprocessing Map, with 10 elements at a time. Once all 10 are complete, we go onto the next 10, and so on.
Now, the problem: The ninth and tenth process are almost ALWAYS slower than the rest. Is there a reason for that? Am I not doing this the most efficient way?
** I won't be able to share the code for this. So do you have any ideas as to why this may be happening?
Thanks in advance.
回答1:
This is due to the way Pool.map distributes the data among the workers. Use chunksize=1 as a parameter, i.e. map(...,..., chunksize=1).
A similar problem is explained in: python multiprocessing map mishandling of last processes
来源：https://stackoverflow.com/questions/35234368/python-multiprocessing-map-takes-longer-to-complete-last-few-processes