Passing sympy lambda to multiprocessing.Pool.map

问题

I want to execute a sympy lambda function in parallel. I don't know:

why it works in parallel although it is a lambda function
why it stops working when I try executing without the pool
why it works if I uncomment the first return in lambdify

And apparently the markdown preprocessor needs a line of text above the code so this is the code:

from multiprocessing import Pool

import sympy
from sympy.abc import x

def f(m):
    return m.lambdify()(1)

class Mult():
    def lambdify(self):
        # return sympy.lambdify(x, 2*x, 'numpy')
        self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
        return self._lambdify

if __name__ == '__main__':
    with Pool() as pool:
        m = Mult()
        print(pool.map(f, [m]))
        print(pool.map(f, [m]))
        print(f(m))
        print(pool.map(f, [m]))

It prints:

[2]
[2]
2
PicklingError: Can't pickle <function <lambda> at 0x000000000DF0D048>: attribute lookup <lambda> on numpy failed

(I cut the traceback)

If I uncomment, it works normally:

[2]
[2]
2
[2]

I tested only on Windows and it works exactly the same with 'numexpr' instead of 'numpy'.

回答1:

The object Mult has no fields when it is created. It can thus be pickled with the stock pickle library. Then, when you call lambdify, you add a _lambdify attribute to the object containing a lambda expression, which cannot be pickled. This causes a failure in the map function

This explains why before calling lambdify you can pickle the object and use Pool.map and why it fails after the call. When you uncomment the line in lambdify, you do not add the attribute to the class, and the Mult object can still be pickled after calling lambdify.

回答2:

Though I have not fully explored this yet, I just want to put on record that the same example works just fine when using loky instead of multiprocessing:

from loky import get_reusable_executor

import sympy
from sympy.abc import x

def f(m):
    return m.lambdify()(1)

class Mult():
    def lambdify(self):
#        return sympy.lambdify(x, 2*x, 'numpy')
        self._lambdify = sympy.lambdify(x, 2 * x, 'numpy')
        return self._lambdify


executor = get_reusable_executor()

m = Mult()
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('pool.map(f, [m])', list(executor.map(f, [m])))
print('f(m)', f(m))
print('pool.map(f, [m])', list(executor.map(f, [m])))

with output

pool.map(f, [m]) [2]
pool.map(f, [m]) [2]
f(m) 2
pool.map(f, [m]) [2]

来源：https://stackoverflow.com/questions/45085095/passing-sympy-lambda-to-multiprocessing-pool-map

标签

python

multiprocessing

sympy