Compiling & installing C executable using python's setuptools/setup.py?

Question

I've got a python module that calls an external binary, built from C source.

The source for that external executable is part of my python module, distributed as a .tar.gz file.

Is there a way of unzipping, then compiling that external executable, and installing it using setuptools/setup.py?

What I'd like to achieve is:

• installing that binary into virtual environments
• manage compilation/installation of the binary using setup.py install, setup.py build etc.
• making the binary part of my python module, so that it can be distributed as a wheel without external dependencies

Answer 1

Solved in the end by modifying setup.py to add additional handlers for commands which did the installation.

An example of a setup.py which does this might be:

import os
from setuptools import setup
from setuptools.command.install import install
import subprocess

def get_virtualenv_path():
    """Used to work out path to install compiled binaries to."""
    if hasattr(sys, 'real_prefix'):
        return sys.prefix

    if hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix:
        return sys.prefix

    if 'conda' in sys.prefix:
        return sys.prefix

    return None


def compile_and_install_software():
    """Used the subprocess module to compile/install the C software."""
    src_path = './some_c_package/'

    # compile the software
    cmd = "./configure CFLAGS='-03 -w -fPIC'"
    venv = get_virtualenv_path()
    if venv:
        cmd += ' --prefix=' + os.path.abspath(venv)
    subprocess.check_call(cmd, cwd=src_path, shell=True)

    # install the software (into the virtualenv bin dir if present)
    subprocess.check_call('make install', cwd=src_path, shell=True)


class CustomInstall(install):
    """Custom handler for the 'install' command."""
    def run(self):
        compile_and_install_software()
        super().run()


setup(name='foo',
      # ...other settings skipped...
      cmdclass={'install': CustomInstall})

Now when python setup.py install is called, the custom CustomInstall class is used, this then compiles and installs software before the normal install steps are run.

You can also do similar for any other steps you're interested in (e.g. build/develop/bdist_egg etc.).

An alternative is to make the compile_and_install_software() function a subclass of setuptools.Command, and create a fully fledged setuptools command for it.

This is more complicated, but lets you do things like specify it as a subcommand of another command (to e.g. avoid executing it twice), and to pass custom options in to it on the command line.