How to make sure that all the python **pool.apply_async()** calls are executed before the pool is closed?

问题

How to make sure that all the pool.apply_async() calls are executed and their results are accumulated through callback before a premature call to pool.close() and pool.join()?

numofProcesses = multiprocessing.cpu_count()
pool = multiprocessing.Pool(processes=numofProcesses)

jobs=[]

for arg1, arg2 in arg1_arg2_tuples:
    jobs.append(pool.apply_async(function1,
                     args=(arg1,
                           arg2,
                           arg3,),
                     callback=accumulate_apply_async_result))

pool.close()
pool.join()

回答1:

You need to wait on the appended AsyncResult objects before exiting the pool. That's

for job in jobs:
    job.wait()

before the pool.close().

But you may be working too hard here. You could

with multiprocessing.Pool() as pool:
    for result in pool.starmap(function1, 
            (arg_1, arg_2, arg_3) for arg_1, arg_2 in sim_chr_tuples)):
        accumulate_apply_async_result(result)

the default for Pool is cpu_count() so no need to add it
with does the close/join for you
starmap waits for results for you

A full working example is

import multiprocessing

result_list = []

def accumulate_apply_async_result(result):
    result_list.append(result)

def function1(arg1, arg2, arg3):
    return arg1, arg2, arg3

sim_chr_tuples = [(1,2), (3,4), (5,6), (7,8)]
arg_3 = "third arg"

if __name__ == "__main__":
    with multiprocessing.Pool() as pool:
        for result in pool.starmap(function1,
                ((arg_1, arg_2, arg_3) 
                for arg_1, arg_2 in sim_chr_tuples)):
            accumulate_apply_async_result(result)

    for r in result_list:
        print(r)

回答2:

In such situation, the better option could be ProcessPoolExecutor.

This is part of concurrent.futures module.

And the module has a function concurrent.futures.as_completed, which can help you in knowing the status.

回答3:

If you're worried that calling close or join will somehow cancel the jobs in progress, and that's why you shouldn't call them "prematurely", you don't need to worry about that. close tells the workers to exit after finishing all work that's been queued, and join waits for them to exit. Together, they are the simplest way to wait for all of your work to complete, so waiting for the work to complete before calling them would be redundant.

来源：https://stackoverflow.com/questions/65350843/how-to-make-sure-that-all-the-python-pool-apply-async-calls-are-executed-b

标签

python

multiprocessing