Pikling error: function MyClass.metho not the same object as __module__.MyClass.metho

和自甴很熟 提交于 2019-12-11 14:36:07

问题


I'm trying to implement metaclass/decorator facilities to allow easy-ish parallelization of code. I'm using Python's multiprocessing.

Say I have:

class Worker(metaclass=Parallelizable):
    def __init__(self):
        super().__init__()

# annotate some method for parallele computation
@ParalleleMethod
def long_calculation(self, data):
    # do stuff
    return ans

class ParalleleMethod:
  def __init__(self, func):
    self.func = func
  def __call__(self, data):
    # as a prototype of the idea I want to get to
    pool.starmap(self.func, data)

When pool.starmap() executes, I get:

_pickle.PicklingError: 
Can't pickle <function Worker.long_calculation at 0x7fdbccd4bea0>: 
it's not the same object as __main__.Worker.long_calculation

I'm guessing that this is the reason:

Note that functions (built-in and user-defined) are pickled by “fully qualified” name reference, not by value. 2 This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised.

Since I passed the function to init as an object, that object isn't fully qualified anymore.

Is there a way that I could "add" that full qualification again to self.func? The assumption being that if I were able to do that, pickling would work - it would find the definition it needs to pickle (and unpickle in the other process)?

来源:https://stackoverflow.com/questions/57104584/pikling-error-function-myclass-metho-not-the-same-object-as-module-myclass

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!