Cython Pickling in Package “not found as” Error

Question

I'm having trouble pickling a Cython class, but only when it's defined inside a package. This problem was noted previously online, but they didn't state how it was resolved. There are two components here: the Cython pickling using a __reduce__ method and a package error.

Cython Pickling Success

I'll first show how it works without the package part. This example works correctly.

Cython File

My Cython file is reudce.pyx:

cdef class Foo(object):
    cdef int n

    def __init__(self, n):
        self.n = n

    def __reduce__(self):
        return Foo, (self.n,)

Setup File

This may be compiled with a setup.py:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("reduce", ["reduce.pyx"])]
)

by executing python setup.py build && cp build/lib*/reduce.so .

Test Script

The test script is called test_reduce.py and is:

import reduce
import pickle
f = reduce.Foo(4)
print pickle.dumps(f)

Executing python test_reduce.py works fine.

Cython Pickling in Package Failure

However, once the reduce.pyx is put into a package, there is an error.

Package Creation

To reproduce this, first create a package called bar.

mkdir bar
mv reduce.so bar
echo "from reduce import Foo" > bar/__init__.py 

Test Script

Change the test_reduce.py file to be:

import bar
import pickle
f = bar.Foo(4)
print pickle.dumps(f)

Error Message

Running python test_reduce.py gives the following error:

File "/usr/lib/python2.7/pickle.py", line 286, in save
  f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python2.7/pickle.py", line 748, in save_global
  (obj, module, name))
pickle.PicklingError: Can't pickle <type 'reduce.Foo'>: it's not found as reduce.Foo

There is a catch of errors which are all turned into a PicklingError in pickle.py After looking at that code, the specific error which is occuring is:

ImportError: No module named reduce

Sanity Test

To check that there is not some kind of scope or other issue, if I run the steps which the pickle module should execute, everything works:

f = bar.Foo(4)
call, args = f.__reduce__()
print call(*args)

So what's going on here?!

Answer 1

The problem was in the build script. The Pickle module uses the __module__ attribute of a function/class for pickling. That __module__ attribute comes from the first argument to the Extension() constructor in the setup.py script. Since I defined my constructor to be Extension('reduce', ['reduce.pyx']), the __module__ attribute is reduce. It should be bar/reduce though since it's now in a package.

Making setup.py look like:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension('bar/reduce', ['reduce.pyx'])]
)

solves the problem.

Answer 2

EDIT

maybe something like this is possible: Foo.__module__ = 'bar.reduce' in bar/__init__.py

Or you can use the module copyreg. Here are some snippets from the code of copyreg:

"""Helper to provide extensibility for pickle.

This is only useful to add pickle support for extension types defined in
C, not for instances of user-defined classes.
"""

def pickle(ob_type, pickle_function, constructor_ob=None):
    # ...

# Example: provide pickling support for complex numbers.

try:
    complex
except NameError:
    pass
else:

    def pickle_complex(c):
        return complex, (c.real, c.imag)

    pickle(complex, pickle_complex, complex)

OLD VERSION

Pickle does this:

    try:
        __import__(module, level=0)
        mod = sys.modules[module]
        klass = getattr(mod, name)
    except (ImportError, KeyError, AttributeError):
        raise PicklingError(
            "Can't pickle %r: it's not found as %s.%s" %
            (obj, module, name))

Can you try out which line fails?

When pickle piclkes classes and functions like module.function it reasssures itself that this is the right function meant:

# module.py
def function(): # 1
    pass
a = function
def function(): # 2
    pass

Function 1 in a can not be pickled but function 2 can be pickled because it is found in the module under the __name__ it has which is "function".

So in your case pickle does not find the same class Foo in the reduce module as is passed as argument. The argument Foo claims to be found in the module reduce.