Printing a Parellel Function Outputs in True Order w/Python

问题

Looking to print everything in order, for a Python parallelized script. Note the c3 is printed prior to the b2 -- out of order. Any way to make the below function with a wait feature? If you rerun, sometimes the print order is correct for shorter batches. However, looking for a reproducible solution to this issue.

from joblib import Parallel, delayed, parallel_backend
import multiprocessing

testFrame = [['a',1], ['b', 2], ['c', 3]]

def testPrint(letr, numbr):
  print(letr + str(numbr))
  return letr + str(numbr)

with parallel_backend('multiprocessing'):
  num_cores = multiprocessing.cpu_count()
  results = Parallel(n_jobs = num_cores)(delayed(testPrint)(letr = testFrame[i][0], 
        numbr = testFrame[i][1]) for i in range(len(testFrame))) 

print('##########')
for test in results:
  print(test)

Output:

b2
c3
a1
##########
a1
b2
c3

Seeking:
a1
b2
c3
##########
a1
b2
c3

回答1:

Once you launch tasks in separate processes you no longer control the order of execution so you cannot expect the actions of those tasks to execute in any predictable order - especially if the tasks can take varying lengths of time.

If you are parallelizing(?) a task/function with a sequence of arguments and you want to reorder the results to match the order of the original sequence you can pass sequence information to the task/function that will be returned by the task and can be used to reconstruct the original order.

If the original function looks like this:

def f(arg):
    l,n = arg
    #do stuff
    time.sleep(random.uniform(.1,10.))
    result = f'{l}{n}'
    return result

Refactor the function to accept the sequence information and pass it through with the return value.

def f(arg):
    indx, (l,n) = arg
    time.sleep(random.uniform(.1,10.))
    result = (indx,f'{l}{n}')
    return result

enumerate could be used to add the sequence information to the sequence of data:

originaldata = list(zip('abcdefghijklmnopqrstuvwxyz', range(26)))
dataplus = enumerate(originaldata)

Now the arguments have the form (index,originalarg) ... (0, ('a',0'), (1, ('b',1)).

And the returned values from the multi-processes look like this (if collected in a list) -

[(14, 'o14'), (23, 'x23'), (1, 'b1'), (4, 'e4'), (13, 'n13'),...]

Which is easily sorted on the first item of each result, key=lambda item: item[0], and the values you really want obtained by picking out the second items after sorting results = [item[1] for item in results].

来源：https://stackoverflow.com/questions/55109219/printing-a-parellel-function-outputs-in-true-order-w-python

标签

python

parallel-processing

python-multiprocessing

joblib