Distutils appharently fails with a (working) SWIG extension

Question

I am wrapping via SWIG a C library in a python module, here called "myExample". If I compile:

$swig -python myExample.i
$gcc -c myExample_wrap.c -I /usr/lib/python2.7 -fPIC -std=c99
$ld -shared myExample_wrap.so -llapacke -o _myExample.so

I obtain a full working module (liblapacke is necessary for some functions I used). Now I'd like to make this module installable via "pip install".

According to the distutils section (https://docs.python.org/2.7/distutils/setupscript.html), I wrote my setup.py file:

from distutils.core import setup, Extension
setup(name='myExample',
version='0.1',
ext_modules=[Extension('_myExample',['myExample.i'], libraries= ['lapacke'])] )

and edited MANIFEST.in in a way to preserve the sources and avoid problems like in similar questions on this website (i.e. just including myExample.h and myExample.c). Then I run:

$python setup.py sdist

and obtained the package installable via "pip install". It seemed done (no errors, no warnings), but...it does not work. In this installable module ("_myExample.so" - notice the underscore, it seems to be required by distutils [maybe does it hide an answer?]) some methods are different, some are missing, etc... Consequently I decided to to one step per time. By just compiling:

$python setup.py build_ext

I already obtained the same problem: the final module is different from the one obtained via the usual compilation explained at the beginning.

Summing up: given a SWIG interface, compiling it traditionally or compiling it via distutils produces a different result. How could it be possible? Is my setup.py wrong? Is there maybe an alternative way for obtaining a pip-installable module without relying on distutils or setuptools(which produces the same problems)?

Ps: the wrap code is very long and so I cannot unfortunately give a detailed list, but I am fully available in adding more in case of need. For instance, the manually compiled interface successfully contains the "AdaptiveInterpolation" (working fine), while the distutil-produced one have "AdaptiveInterpolation_set", "AdaptiveInterpolation_get", or there are a lot of methods starting with "new_" (absent in my original code).

Answer 1

Practically, there are two options for distribution of such package: bdist_wheel and sdist.

Let's take SWIG's example from the docs for distutils.

example.h

int fact(int n);

example.c

#include "example.h"

int fact(int n) {
    if (n == 0) {
        return 1;
    }
    else {
        return n * fact(n - 1);
    }
}

example.i

%module example

%{
#define SWIG_FILE_WITH_INIT
#include "example.h"
%}

int fact(int n);

Let's build a SWIG interface, which by default produces example_wrap.c.

swig -python example.i 

Wheel

Python Wheels is a modern distribution format which simplifies distribution of pre-built packages (i.e. users of the package will not need a compiler, development headers and the like to install it). To use it we'll need setuptools and wheel (can be installed with pip or from OS repositories, sudo apt-get install python-wheel python-setuptools).

setup.py

from setuptools import setup, Extension


setup(
  name = 'example',
  version = '0.1',
  author = 'SWIG Docs',
  description = 'Simple swig example from docs',
  ext_modules = [
    Extension('_example', sources = ['example_wrap.c', 'example.c'])],
  py_modules = ['example'],
  package_data = {'': ['example.h']}  # needed for sdist
)

You can build the wheel with:

python setup.py bdist_wheel

On my machine it produces example-0.1-cp27-cp27mu-linux_x86_64.whl which I can install with pip and test like python -c 'import example; print(example.fact(5))'. Note the filename. It encodes compatible Python version, ABI and platform. Here's listing of the contents (unzip -l ...).

Archive:  example-0.1-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     2609  2018-02-24 11:06   example.py
    70968  2018-02-24 13:31   _example.so
       10  2018-02-24 14:58   example-0.1.dist-info/DESCRIPTION.rst
      290  2018-02-24 14:58   example-0.1.dist-info/metadata.json
       17  2018-02-24 14:58   example-0.1.dist-info/top_level.txt
      105  2018-02-24 14:58   example-0.1.dist-info/WHEEL
      193  2018-02-24 14:58   example-0.1.dist-info/METADATA
      617  2018-02-24 14:58   example-0.1.dist-info/RECORD
---------                     -------
    74809                     8 files

To build wheels with better compatibility, you can, for instance, look at manylinux.

Source

sdist stands for source distribution, which means your users will need to have a compiler and relevant development dependencies. Source distribution is built with:

python setup.py sdist

Produced example-0.1.tar.gz contains (tar -tf ...):

example-0.1/
example-0.1/example.c
example-0.1/PKG-INFO
example-0.1/example.egg-info/
example-0.1/example.egg-info/PKG-INFO
example-0.1/example.egg-info/dependency_links.txt
example-0.1/example.egg-info/top_level.txt
example-0.1/example.egg-info/SOURCES.txt
example-0.1/setup.cfg
example-0.1/example.py
example-0.1/setup.py
example-0.1/example_wrap.c
example-0.1/example.h