How to implement custom control over python multiprocessing.Pool?

问题

Usually i use following code, and it works fine when you do not matter in which order function process_func will handle some parameter:

params = [1,2,3,4,5 ... ]

def process_func():
    ...

pool = new Pool(40)
pool.map(process_func, params)
pool.close()
pool.join()

In example above we have processes of one type, with maximum simultanious number of 40. But.. imagine we have processes (parameters) of different type, which should be executed simultaniously. For example, in my selenium grid i have 40 firefox, 40 chromes. And i have 5000 test cases, some of them prefer chrome, some of them firefox, some of them does not matter.

For example, lets say we have following types:

• type firefox: maximum simultanious number: 40
• type chrome: maximum simultanious number: 40

In this case our pool will have maximum of 80 simultanious processes, but there is strict rule: 40 of them must be firefox, 40 of them must be chromes.

It means that params won't be taken one after each other. Pool must select value from params list in a way to have maximum of each process type.

How it is possible to achieve that?

回答1:

I would modify your process_func() to take one more parameter that tells it which "type" to be and use two separate pools. Adding functools.partial will allow us to still use pool.map():

from functools import partial
from multiprocessing import Pool

params = [1,2,3,4,5 ... ]

def process_func(type, param):
    if type == 'Firefox':
        # do Firefox stuff
    else:
        # do Chrome stuff

chrome_pool = Pool(40)
fox_pool = Pool(40)

chrome_function = partial(process_func, 'Chrome')
fox_function = partial(process_func, 'Firefox')

chrome_pool.map(chrome_func, params)
fox_pool.map(fox_func, params)

chrome_pool.close()
fox_pool.close()
chrome_pool.join()
fox_pool.join()

The functools.partial() function allows us to bind an argument to a specific value, by returning a new function object that will always supply that argument. This approach allows you to limit each "type" (for lack of a better term) to 40 worker processes.

来源：https://stackoverflow.com/questions/26534540/how-to-implement-custom-control-over-python-multiprocessing-pool