When pickling a class I get different behavior in python that in cython

问题

I have the following file hierarchy:

python/apps/A.py
      /geometrylib/__init__.py
      /geometrylib/B.py
      /geometrylib/geometry.py
      /geometrylib/goemetry.pyx
      /geometrylib/goemetry.pyd

geometry.pyx and geometry.py contain the same class Camera (the cython version defines the class with cdef ). Both A.py and B.py import the geometry module.

If I import the cython version(compiled to geometry.pyd), I can correctly pickle Camera from within B.py in the python/geometrylib folder. But I can't pickle Camera from A.py in the python/apps folder, I get the following exception:

pickle.PicklingError: Can't pickle : it's not found as geometry.Camera

However, if I delete the geometry.pyd and I import the python version(geometry.py) instead, then I can pickle Camera from either A.py or B.py. Nothing else changes apart from deleting geometry.pyd, same python command line, run from the same folder in both cases. why this difference?

Digging a bit I see that the exception occurs in C:\Python27\Lib\pickle.py line 742

try:
    __import__(module)            #line 742
    mod = sys.modules[module]
    klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError):
    raise PicklingError(
        "Can't pickle %r: it's not found as %s.%s" %
        (obj, module, name))

When in A.py I import the cython version(geometry.pyd), (and I pickle a Camera instance to trigger the expection) module is "geometry" and __import__(module) triggers the exception. When in A.py I import the python version(geometry.py),(and I pickle a Camera instance to trigger the expection) module is "geometrylib.geometry" and __import__(module) imports the module correctly.

I have solved the problem by adding python/geometrylib to PYTHONPATH, then I can correctly pickle Camera from both A.py and B.py using the cython version.

Is this how is it supposed to work? I don't like my solution. Does anybody have a better solution?

EDITED to add some extra information.

Also, by request, this is the setup.py I used to build the cython extension.

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy

setup(
    cmdclass = { 'build_ext': build_ext},
    ext_modules = [Extension("geometry", ['geometry.pyx'], include_dirs=[numpy.get_include(), "."])])

回答1:

You are building geometry.pyx as a top-level module, while in fact it is part of geometrylib package. As a result, Camera class gets assigned incorrect __module__ value (geometry instead of geometrylib.geometry) and pickler fails when it tries to find top-level module named geometry.

You should follow standard packaging guidelines, i.e. put setup.py into "Python" folder, next to the top-level module (geometrylib). setup call will look like this:

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("geometrylib.geometry", ['geometrylib/geometry.pyx'], include_dirs=[numpy.get_include(), "."])])

来源：https://stackoverflow.com/questions/12941899/when-pickling-a-class-i-get-different-behavior-in-python-that-in-cython

标签

python

import

pickle

cython