Building a module with setuptools and swig

问题

I have a bunch of SWIG interfaces (foo.i, bar.i, etc.). I want to build them into a Python (3.6.4) module for my platform (MS Windows), with setuptools. The module should include the SWIG-generated Python files (*.py), the binary extensions (*.pyd) and the compiled caches (*.pyc). My setup.py is essentially like this:

from setuptools import setup, Extension
from pathlib import Path
paths=list(Path('.').glob('*.i'))
py=[path.stem for path in paths]
ext=[Extension('_' + path.stem, [str(path)]) for path in paths]
setup(py_modules=py, ext_modules=ext)

Now I build it with the following steps:

python setup.py build_ext -I..\include --swig-opts="-I..\include -c++" -b pkg
python setup.py build_py -c -d pkg
echo. > pkg\__init__.py

Using these steps, I get exactly what I want, under the pkg directory.

My question is: Is there no way to get this effect using a single invocation of setup.py, e.g., setup.py build? I think that build is supposed to call build_ext, but then I cannot see how to pass, e.g., the swig-opts option.

Update

Passing the SWIG options is solved (h/t @hoefling). The solution looks like this:

ext=[Extension(name='_' + path.stem,
            sources=[str(path)],
            swig_opts=['-I../include', '-c++'],
            include_dirs=['../include'])
            for path in paths]

However, with that layer of the onion peeled off, I can now see the layer below, which is this: setup.py build as a single invocation, wants to run the build_py first, and then build_ext afterward. Can you see why this fails? I have no Python sources. The Python scripts in my module are to be generated by SWIG, but SWIG doesn't get run until the build_ext step. Thus, I end up with the same question, which is how to build the module in a single invocation.

C:\some\path> python setup.py build
running build
running build_py
file foo.py (for module foo) not found
file foo.py (for module foo) not found
running build_ext
building '_foo' extension
swigging foo.i to foo_wrap.cpp
swig.exe -python -I../include -c++ -o foo_wrap.cpp foo.i
creating build
creating build\temp.win-amd64-3.6
creating build\temp.win-amd64-3.6\Release
cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I../include /EHsc /Tpfoo_wrap.cpp /Fobuild\temp.win-amd64-3.6\Release\foo_wrap.obj foo_wrap.cpp
Creating build\wib.win-amd64-3.6
link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /EXPORT:PyInit__foo build\temp.win-amd64-3.6\Release\foo_wrap.obj /IMPLIB:build\temp.win-amd64-3.6\Release\_foo.cp36-win_amd64.lib
Creating library build\temp.win-amd64-3.6\Release\_foo.cp36-win_amd64.lib and object build\temp.win-amd64-3.6\Release\_foo.cp36-win_amd64.exp
Generating code
Finished generating code

When complete, foo.py exists in the current working directory next to foo.i, and _foo.cp36-win_amd64.pyd exists under build\lib.win-amd64-3.6. (FYI, this transcript is edited slightly to protect proprietary information, e.g., I don't show all the full paths to all the echos of commands. And I did not include bar.i in the sources.)

To clarify:

The error is this one: File foo.py (for module foo) not found
Though all the required files exist in the output, except foo.cp36-win_amd64.pyc which I could possibly live without, I have to manually copy foo.py and _foo.cp36-win_amd64.pyd into some new directory to make a clean Python module.

回答1:

The problem is that once you declare py_modules, distutils includes build_py as subcommand in build and it is always being executed first because of the ordering of sub_commands list in build class. It simply doesn't know that SWIG will generate python module wrappers later in build_ext. I guess you can see this as a bug since one would expect distutils to be able to cope with generated modules, but meh.

Since you only have SWIG interfaces and all of your python modules are generated wrappers (so in your case build_ext is not dependent on build_py), you can change the ordering of subcommands in build so build_ext is executed first, then the rest. This will ensure that the python modules are generated before build_py is executed.

Basically, you need override the build command class and change the ordering of elements in sub_commands list. There are lots of ways to reorder a list in python; below is my proposal. For reordering the list I used the list conditional partitioning recipe from itertools recipes:

import itertools
from setuptools import setup, Extension
from distutils.command.build import build as build_orig


def partition(pred, iterable):
    t1, t2 = itertools.tee(iterable)
    return itertools.filterfalse(pred, t1), filter(pred, t2)


class build(build_orig):

    def finalize_options(self):
        super().finalize_options()
        condition = lambda el: el[0] == 'build_ext'
        rest, sub_build_ext = partition(condition, self.sub_commands)
        self.sub_commands[:] = list(sub_build_ext) + list(rest)


setup(
    ...,
    cmdclass={'build': build},
)

回答2:

Since the essential idea (see @hoefling's answer) is to subclass the build command to set its sub_commands property, this works:

from setuptools import setup, Extension
from distutils.command.build import build

class build_alt_order(build):
  def __init__(self, *args):
    super().__init__(*args)
    self.sub_commands = [('build_ext', build.has_ext_modules),
                         ('build_py', build.has_pure_modules)]

setup(py_modules=['foo.py', ...],
      ext_modules=[Extension('_foo', swig_opts=...), ...],
      cmdclass={'build':build_alt_order})

来源：https://stackoverflow.com/questions/50239473/building-a-module-with-setuptools-and-swig

标签

python

python-3.x

swig

setuptools