Distributing a shared library and some C code with a Cython extension module

Question

I'm trying to take some functions from a large C++ shared library (libbig.so) and expose them to Python via Cython. To do so, I've got a little C++ file (small.cpp) that provides a thin wrapper around the functionality from the shared library that I need, in a way that makes it easy to call via Cython (pysmall.pyx).

libbig.so -> small.cpp, small.h -> libsmall.so -> pysmall.pyx -> pysmall.cpp -> pysmall.so

I can build and run this extension module on my own computer: I just compile small.cpp into libsmall.so, and then say "libraries=['small']" in the Extension object in setup.py to build the extension module pysmall.so.

I'm now trying to distribute this extension module, and I am having a hard time tracking down resources that describe setup.py best practices for distributing a Cython module as well as C source and shared libraries. I've read through "Installing Python Modules", "Distributing Python Modules", and "Distributing Cython Modules". I understand how to distribute an extension module on its own. I'm less certain about the best way to distribute the dependencies of the extension module.

The Cython documentation indicates that you should include the generated .cpp files as well as the .pyx files, in case Cython is not present, yet it does not provide code to demonstrate how to best handle each situation. It also does not mention how to distribute the shared libraries on which the Cython module depends.

I'm digging through the setup.py scripts from pandas, lxml, pyzmq, h5py, and more, and there's quite a bit of extraneous work happening. If anyone has pointers or example code that might accelerate this process, I'd certainly appreciate it!

Answer 1

1) Distributing libbig.so

This is a problem that python isn't going to help you with. Who are you targeting? If it's linux, can you request that they install it with their package manager? If libbig isn't distributed through a package manager or it's not linux and you're targeting multiple architectures, you might have to distribute the libbig source.

2) Cython/setuptools.

Frankly, I think its easiest to just require that people have Cython. This way there's only one ground truth version of the code, and you don't have to worry about inconsistencies between the .pyx and .cpp code. The easiest way to do this is to use setuptools instead of distutils. That way, you can use:

setup('mypackage',
    ...
    install_requires=['cython'])

In total, your setup.py script will look something like:

# setup.py

from setuptools import setup, Extension
from Cython.Distutils import build_ext

pysmall = Extension('pysmall',
    sources = ['pysmall.pyx', 'small.cpp'],
    include_dirs = ['include/'])

setup(name='mypackage',
      packages=['yourpurepythonpackage'],
      install_requires=['cython==0.17'],
      ext_modules=[pysmall],
      cmdclass = {'build_ext': build_ext})

If you don't like the idea of requiring cython, you could do something like:

# setup.py

import warnings
try:
    from Cython.Distutils import build_ext
    from setuptools import setup, Extension
    HAVE_CYTHON = True
except ImportError as e:
    HAVE_CYTHON = False
    warnings.warn(e.message)
    from distutils.core import setup, Extension
    from distutils.command import build_ext

pysmall = Extension('pysmall',
    sources = ['pysmall.pyx', 'small.cpp'],
    include_dirs = ['include/'])

configuration = {'name': 'mypackage',
      'packages': ['yourpurepythonpackage'],
      'install_requires': ['cython==0.17'],
      'ext_modules': [pysmall],
      'cmdclass': {'build_ext': build_ext}}

if not HAVE_CYTHON:
    pysmall.sources[0] = 'pysmall.cpp'
    configuration.pop('install_requires')

setup(**configuration)

Answer 2

Here's my tricky solution. The idea is to "hide" the presence of cython until it's installed by the requirements. This can be achieved by lazy evaluation. Here's an example:

from setuptools import setup, Extension

class lazy_cythonize(list):
    def __init__(self, callback):
        self._list, self.callback = None, callback
    def c_list(self):
        if self._list is None: self._list = self.callback()
        return self._list
    def __iter__(self):
        for e in self.c_list(): yield e
    def __getitem__(self, ii): return self.c_list()[ii]
    def __len__(self): return len(self.c_list())

def extensions():
    from Cython.Build import cythonize
    ext = Extension('native_ext_name', ['your/src/*.pyx'])
    return cythonize([ext])


configuration = {
    'name': 'mypackage',
    'packages': ['yourpurepythonpackage'],
    'install_requires': ['cython==0.17'],
    'ext_modules': lazy_cythonize(extensions)
}

setup(**configuration)

lazy_cythonize is a fake list that generates its internal elements only when someone tries to access to it.
When it's required, this class imports Cython.Build and generates the list of extensions. This avoids to keep the *.c files in your project, requiring cython to be installed when the module is building.

Quite tricky, but actually it's working.

Answer 3

I've pushed a fix for setuptools 288, slated for release as setuptools 18.0. This changelog entry describes a technique that should work with that build. A beta release is available for testing.