How can I recover the return value of a function passed to multiprocessing.Process?

前端 未结 12 1781
野的像风
野的像风 2020-11-22 07:36

In the example code below, I\'d like to recover the return value of the function worker. How can I go about doing this? Where is this value stored?

12条回答
  •  温柔的废话
    2020-11-22 07:54

    For some reason, I couldn't find a general example of how to do this with Queue anywhere (even Python's doc examples don't spawn multiple processes), so here's what I got working after like 10 tries:

    def add_helper(queue, arg1, arg2): # the func called in child processes
        ret = arg1 + arg2
        queue.put(ret)
    
    def multi_add(): # spawns child processes
        q = Queue()
        processes = []
        rets = []
        for _ in range(0, 100):
            p = Process(target=add_helper, args=(q, 1, 2))
            processes.append(p)
            p.start()
        for p in processes:
            ret = q.get() # will block
            rets.append(ret)
        for p in processes:
            p.join()
        return rets
    

    Queue is a blocking, thread-safe queue that you can use to store the return values from the child processes. So you have to pass the queue to each process. Something less obvious here is that you have to get() from the queue before you join the Processes or else the queue fills up and blocks everything.

    Update for those who are object-oriented (tested in Python 3.4):

    from multiprocessing import Process, Queue
    
    class Multiprocessor():
    
        def __init__(self):
            self.processes = []
            self.queue = Queue()
    
        @staticmethod
        def _wrapper(func, queue, args, kwargs):
            ret = func(*args, **kwargs)
            queue.put(ret)
    
        def run(self, func, *args, **kwargs):
            args2 = [func, self.queue, args, kwargs]
            p = Process(target=self._wrapper, args=args2)
            self.processes.append(p)
            p.start()
    
        def wait(self):
            rets = []
            for p in self.processes:
                ret = self.queue.get()
                rets.append(ret)
            for p in self.processes:
                p.join()
            return rets
    
    # tester
    if __name__ == "__main__":
        mp = Multiprocessor()
        num_proc = 64
        for _ in range(num_proc): # queue up multiple tasks running `sum`
            mp.run(sum, [1, 2, 3, 4, 5])
        ret = mp.wait() # get all results
        print(ret)
        assert len(ret) == num_proc and all(r == 15 for r in ret)
    

提交回复
热议问题