pathos multiprocessing cannot pickle

Question

I am having a similar issue to this person. I am unable to run a simple multiprocessing routine in the pathos module and receive a pickling error. Below is the code and error.

from pathos.multiprocessing import ProcessingPool
import dill
class ProcClass(object):
    def __init__(self):
        pass
    def f(self,x):
        return x*x
pc = ProcClass()
pl = ProcessingPool(3)
print pl.map(pc.f, range(10))

The returned error:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Library/Frameworks/EPD64.framework/Versions/7.3/lib/python2.7/multiprocessing/pool.py", line 320, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

I have installed: pp, dill, pox and multiprocessing then installed pathos. The install works but always gives me this error:

---

WARNING: One of the following dependencies is unresolved: pp(ft) >=1.6.4.5 dill >=0.2.4 pox >=0.2.2 (multi)processing

---

Based on a response from the author of pathos to a similar question, it looks like there is a problem with the install. I have removed and reinstalled several times, each time verifying the proper dependancies are installed. I'm running on MacOS and using python 2.7. Any help will be greatly appreciated!

Answer 1

It appears that you are missing a critical dependency. With pathos, you should use multiprocess, which is a fork of multiprocessing which uses the dill serializer.

>>> import multiprocessing
>>> multiprocessing.__version__
'0.70a1'
>>> import multiprocess   
>>> multiprocess.__version__
'0.70.4.dev0'
>>> 
>>> multiprocess.Pool().map(lambda x:x*x, range(10))
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>>