Is it possible to multiprocess a function that returns something in Python?

前端 未结 6 1413
轻奢々
轻奢々 2020-12-03 02:54

In Python I have seen many examples where multiprocessing is called but the target just prints something. I have a scenario where the target returns 2 variables, which I nee

6条回答
  •  暖寄归人
    2020-12-03 03:35

    Why nobody uses callback of multiprocessing.Pool?

    Example:

    from multiprocessing import Pool
    from contextlib import contextmanager
    
    from pprint import pprint
    from requests import get as get_page
    
    @contextmanager
    def _terminating(thing):
        try:
            yield thing
        finally:
            thing.terminate()
    
    def _callback(*args, **kwargs):
        print("CALBACK")
        pprint(args)
        pprint(kwargs)
    
    print("Processing...")
    with _terminating(Pool(processes=WORKERS)) as pool:
        results = pool.map_async(get_page, URLS, callback=_callback)
    
        start_time = time.time()
        results.wait()
        end_time = time.time()
        print("Time for Processing: %ssecs" % (end_time - start_time))
    

    Here, I print both args and kwargs. But you can replace callback by:

    def _callback2(responses):
        for r in responses:
            print(r.status_code) # or do whatever with response...
    

提交回复
热议问题