In Python I have seen many examples where multiprocessing is called but the target just prints something. I have a scenario where the target returns 2 variables, which I nee
Why nobody uses callback of multiprocessing.Pool?
Example:
from multiprocessing import Pool
from contextlib import contextmanager
from pprint import pprint
from requests import get as get_page
@contextmanager
def _terminating(thing):
try:
yield thing
finally:
thing.terminate()
def _callback(*args, **kwargs):
print("CALBACK")
pprint(args)
pprint(kwargs)
print("Processing...")
with _terminating(Pool(processes=WORKERS)) as pool:
results = pool.map_async(get_page, URLS, callback=_callback)
start_time = time.time()
results.wait()
end_time = time.time()
print("Time for Processing: %ssecs" % (end_time - start_time))
Here, I print both args and kwargs. But you can replace callback by:
def _callback2(responses):
for r in responses:
print(r.status_code) # or do whatever with response...