Pass multiple parameters to concurrent.futures.Executor.map?

The concurrent.futures.Executor.map takes a variable number of iterables from which the function given is called. How should I call it if I have a generator that produces tuples that are normally unpacked in place?

The following doesn't work because each of the generated tuples is given as a different argument to map:

args = ((a, b) for (a, b) in c)
for result in executor.map(f, *args):
    pass

Without the generator, the desired arguments to map might look like this:

executor.map(
    f,
    (i[0] for i in args),
    (i[1] for i in args),
    ...,
    (i[N] for i in args),
)

vz0

You need to remove the * on the map call:

args = ((a, b) for b in c)
for result in executor.map(f, args):
    pass

This will call f, len(args) times, where f should accept one parameter.

If you want f to accept two parameters you can use a lambda call like:

args = ((a, b) for b in c)
for result in executor.map(lambda p: f(*p), args):   # (*p) does the unpacking part
    pass

One argument that is repeated, one argument in `c`

from itertools import repeat
for result in executor.map(f, repeat(a), c):
    pass

Need to unpack items of `c`, and can unpack `c`

from itertools import izip
for result in executor.map(f, *izip(*c)):
    pass

Need to unpack items of `c`, can't unpack `c`

Change f to take a single argument and unpack the argument in the function.
If each item in c has a variable number of members, or you're calling f only a few times:
```
executor.map(lambda args, f=f: f(*args), c)
```
It defines a new function that unpacks each item from c and calls f. Using a default argument for f in the lambda makes f local inside the lambda and so reduces lookup time.

If you've got a fixed number of arguments, and you need to call f a lot of times:

from collections import deque
def itemtee(iterable, n=2):
    def gen(it = iter(iterable), items = deque(), next = next):
        popleft = items.popleft
        extend = items.extend
        while True:
            if not items:
                extend(next(it))
            yield popleft()
    return [gen()] * n

executor.map(f, *itemtee(c, n))

Where n is the number of arguments to f. This is adapted from itertools.tee.

You can use currying to create new function via partial method in Python

from concurrent.futures import ThreadPoolExecutor
from functools import partial


def some_func(param1, param2):
    # some code

# currying some_func with 'a' argument is repeated
func = partial(some_func, a)
with ThreadPoolExecutor() as executor:
    executor.map(func, list_of_args):
    ...

If you need to pass more than one the same parameters you can pass them to partial method

func = partial(some_func, a, b, c)

For ProcessPoolExecutor.map():

Similar to map(func, *iterables) except:

the iterables are collected immediately rather than lazily;

func is executed asynchronously and several calls to func may be made concurrently.

Try running the following snippet under python 3, and you will be quite clear:

from concurrent.futures import ProcessPoolExecutor

def f(a, b):
    print(a+b)

with ProcessPoolExecutor() as pool:
    pool.map(f, (0, 1, 2, 3, 4, 5, 6, 7, 8, 9), (0, 1, 2))

# 0, 2, 4

array = [(i, i) for i in range(3)]
with ProcessPoolExecutor() as pool:
    pool.map(f, *zip(*array))

# 0, 2, 4

来源：https://stackoverflow.com/questions/6785226/pass-multiple-parameters-to-concurrent-futures-executor-map

标签

python