ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool

Question

I was experimenting with the new shiny concurrent.futures module introduced in Python 3.2, and I\'ve noticed that, almost with identical code, using the Pool from concurrent

Answer 1

When using map from concurrent.futures, each element from the iterable is submitted separately to the executor, which creates a Future object for each call. It then returns an iterator which yields the results returned by the futures.
Future objects are rather heavyweight, they do a lot of work to allow all the features they provide (like callbacks, ability to cancel, check status, ...).

Compared to that, multiprocessing.Pool has much less overhead. It submits jobs in batches (reducing IPC overhead), and directly uses the result returned by the function. For big batches of jobs, multiprocessing is definitely the better options.

Futures are great if you want to sumbit long running jobs where the overhead isn't that important, where you want to be notified by callback or check from time to time to see if they're done or be able to cancel the execution individually.

Personal note:

I can't really think of much reasons to use Executor.map - it doesn't give you any of the features of futures - except for the ability to specify a timeout. If you're just interested in the results, you're better off using one of multiprocessing.Pool's map functions.