How to use setuptools packages and ext_modules with the same name?

Question

I got the following file structure for my Python C Extension project:

.
├── setup.py
├── source
    ├── cppimplementation
    │   └── fastfile.cpp
    └── fastfilepackage
        ├── __init__.py
        └── version.py

And I use the following setup.py file:

from setuptools import setup, Extension

setup(
        name= 'fastfilepackage',
        version= '0.1.1',
        package_dir = {
            '': 'source',
        },

        packages = [
            'fastfilepackage',
        ],

        ext_modules= [
            Extension(
                'fastfilepackage',
                [
                    'source/cppimplementation/fastfile.cpp',
                ]
            )
        ],
    )

I install them with:

$ pip3 --version
pip 19.1.1 (python 3.6)

$ python3 --version
Python 3.6.7

$ pip3 list
Package                Version      
---------------------- -------------
wheel                  0.33.1        
setuptools             40.8.0        
...

fastfilepackage$ pip3 install .

The problem is that when I install it, my Python C Extension module is overridden by fastfilepackage/version.py and fastfilepackage/__init__.py, i.e, after installing it, I got the following:

import fastfilepackage
print( dir( fastfilepackage ) )
# prints ['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', 
#         '__name__', '__package__', '__path__', '__spec__', '__version__',
#         'version']

i.e., no FastFile class exported by source/cppimplementation/fastfile.cpp, but it has the fastfilepackage/version.py and fastfilepackage/__init__.py files.

This is the installed file structure:

.
└── dist-packages
    ├── fastfilepackage
    │   ├── __init__.py
    │   ├── __pycache__
    │   │   ├── __init__.cpython-36.pyc
    │   │   └── version.cpython-36.pyc
    │   └── version.py
    ├── fastfilepackage-0.1.1.dist-info
    │   ├── INSTALLER
    │   ├── LICENSE.txt
    │   ├── METADATA
    │   ├── RECORD
    │   ├── top_level.txt
    │   └── WHEEL
    └── fastfilepackage.cpython-36m-x86_64-linux-gnu.so

But if I remove the lines package_dir = { '': 'source', }, and packages = [ 'fastfilepackage', ], from my setup file, then, my Python C Extension module is correctly installed:

import fastfilepackage
print( dir( fastfilepackage ) )
# prints ['FastFile', '__doc__', '__file__', '__loader__', '__name__', 
#         '__package__', '__spec__']

i.e., it has the FastFile class exported by source/cppimplementation/fastfile.cpp, but it does not have the fastfilepackage/version.py and fastfilepackage/__init__.py files.

This was the installed file structure:

.
└── dist-packages
    ├── fastfilepackage-0.1.1.dist-info
    │   ├── INSTALLER
    │   ├── LICENSE.txt
    │   ├── METADATA
    │   ├── RECORD
    │   ├── top_level.txt
    │   └── WHEEL
    └── fastfilepackage.cpython-36m-x86_64-linux-gnu.so

How can I put ext_modules and packages to use the same package name under my setup.py without one overriding the other?

Answer 1

You cannot. The first one imported wins. You cannot have scripts/modules/packages/extensions with the same name — one overrides all others.

But you can to have one inside another. Make your extension named fastfilepackage.fastfilepackage and you can import fastfilepackage to import the Python package and import fastfilepackage.fastfilepackage to import the extension; or from fastfilepackage import fastfilepackage.

Answer 2

As a final solution, I completely removed all Python *.py code because they caused the C Extensions code to become 30% slower. Now my setup.py become like this:

from setuptools import setup, Extension

setup(
        name = 'fastfilepackage',
        version = '0.1.1',

        ext_modules = [
            Extension(
                name = 'fastfilepackage',
                sources = [
                    'source/fastfile.cpp',
                ],
                include_dirs = ['source'],
            )
        ],
    )

File structure:

.
├── setup.py
├── MANIFEST.in
├── README.md
├── LICENSE.txt
└── source
    ├── fastfile.cpp
    └── version.h

MANIFEST.in

include README.md
include LICENSE.txt

recursive-include source *.h

This is the installed file structure: (No *.py files anywhere = 100% performance)

.
└── dist-packages
    ├── fastfilepackage-0.1.1.dist-info
    │   ├── INSTALLER
    │   ├── LICENSE.txt
    │   ├── METADATA
    │   ├── RECORD
    │   ├── top_level.txt
    │   └── WHEEL
    └── fastfilepackage.cpython-36m-x86_64-linux-gnu.so

I just replaced the version.py directly by a C Extensions module attribute:

// https://docs.python.org/3/c-api/arg.html#c.Py_BuildValue
const char* __version__ = "0.1.1";
PyObject_SetAttrString( thismodule, "__version__", Py_BuildValue( "s", __version__ ) );

References:

1. Define a global in a Python module from a C API
2. Original commit: Deprecated the Python implementation *.py