How to fix or reorganize this multiprocessing pattern to avoid pickling errors?

问题

Another pickling question ... The following leads to pickling errors. I think it is to do with the scoping or something. Am not sure yet.

The goal is to have a decorator that takes arguments and enriches a function with methods. If the best way is to simply construct classes explicitly then that is fine but this is meant to hide things from users writing "content".

import concurrent.futures
import functools

class A():
    def __init__(self, fun, **kwargs):
        self.fun = fun
        self.stuff = kwargs
        functools.update_wrapper(self, fun)
    def __call__(self, *args, **kwargs):
        print(self.stuff, args, kwargs)
        return self.fun(*args, **kwargs)

def decorator(**kwargs):
    def inner(fun):
        return A(fun, **kwargs)
    return inner

@decorator(a=1, b=2)
def f():
    print('f called')

executor = concurrent.futures.ProcessPoolExecutor(max_workers=10)

tasks = [f for x in range(10)]
fut = list()
for task in tasks:
    fut.append(executor.submit(task))
res = [x.result() for x in fut]
print(res)

Error is:

_pickle.PicklingError: Can't pickle <function f at 0x7fe37da121e0>: it's not the same object as __main__.f

回答1:

I ended up doing something like this:

def dill_wrapped(dilled, *args, **kwargs):
    fun = dill.loads(dilled)
    return wrapped(fun, *args, **kwargs)

You should probably try hard to avoid the need for this but you really do need it sometimes when decorating functions.

来源：https://stackoverflow.com/questions/61490167/how-to-fix-or-reorganize-this-multiprocessing-pattern-to-avoid-pickling-errors