How to compile __init__.py file using cython on Windows

Question

When I compile an arbitrary __init__.py file on Windows with setup.py build_ext --inplace command, it has an unresolvable external symbol error (i.e. "LINK

Answer 1

Maybe this behaviour might be viewed as a small bug in distutils-package (as pointed out by @DavidW there is this open issue: https://bugs.python.org/issue35893). However, it also shows, that cythonizing/compiling __init__.py isn't very popular and uses some undocumented implementation details which might change in the future, so it could be wiser to refrain from meddling with __init__.py.

But if you must...

---

When a package is imported explicitly, e.g.

import ctest

or implicitly, e.g.

import ctest.something

The FileFinder will see that a package, and not a module, is imported and will try to load ctest/__init__.py instead of ctest.py (which most likely doesn't exists):

    # Check if the module is the name of a directory (and thus a package).
    if cache_module in cache:
        base_path = _path_join(self.path, tail_module)
        for suffix, loader_class in self._loaders:
            init_filename = '__init__' + suffix
            full_path = _path_join(base_path, init_filename)
            if _path_isfile(full_path):
                return self._get_spec(loader_class, fullname, full_path, [base_path], target)

Used suffix, loader_class are for loading __init__.so, __init__.py and __init__.pyc in this order (see also this SO-post). This means, __init__.so will be loaded instead of __init__.py if we manage to create one.

While __init__.py is executed, The property __name__ is the name of the package, i.e. ctest in your case, and not __init__ as one might think. Thus, the name of the init-function, Python-interpreter will call when loading the extension __init__.so is PyInit_ctest in your case (and not PyInit___init__ as one might think).

The above explains, why it all works on Linux out-of-the-box. What about Windows?

The loader can only use symbols from a so/dll which aren't hidden. Per default all symbols built with gcc are visible, but not for VisualStudio on Windows - where all symbols are hidden per default (see e.g. this SO-post).

However, the init-function of a C-extension must be visible (and only the init-function) so it can be called with help of the loader - the solution is to export this symbol (i.e. PyInit_ctest) while linking, in your case it is the wrong /EXPORT:PyInit___init__-option for the linker.

The problem can be found in distutils, or more precise in build_ext-class:

def get_export_symbols(self, ext):
    """Return the list of symbols that a shared extension has to
    export.  This either uses 'ext.export_symbols' or, if it's not
    provided, "PyInit_" + module_name.  Only relevant on Windows, where
    the .pyd file (DLL) must export the module "PyInit_" function.
    """
    initfunc_name = "PyInit_" + ext.name.split('.')[-1]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

Here, sadly ext.name has __init__ in it.

From here, one possible solution is easy : to override get_export_symbols, i.e. to add the following to your setup.py-file (read on for a even simpler version):

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    names = ext.name.split('.')
    if names[-1] != "__init__":
        initfunc_name = "PyInit_" + names[-1]
    else:
        # take name of the package if it is an __init__-file
        initfunc_name = "PyInit_" + names[-2]
    if initfunc_name not in ext.export_symbols:
        ext.export_symbols.append(initfunc_name)
    return ext.export_symbols

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

Calling python setup.py build_ext -i should be enough now (because __init__.so will be loaded rather than __init__.py).

---

However, as @DawidW has pointed out, Cython uses macro PyMODINIT_FUNC, which is defined as

#define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*

with Py_EXPORTED_SYMBOL being marked as visible/exported on Windows:

#define Py_EXPORTED_SYMBOL __declspec(dllexport)

Thus, there is no need to mark the symbol as visible at the command line. Even worse, this is the reason for the warning LNK4197:

__init__.obj : warning LNK4197: export 'PyInit_ctest' specified multiple times; using first specification

as PyInit_test is marked as __declspec(dllexport) and exported via option /EXPORT: at the same time.

/EXPORT:-option will be skipped by distutils, if export_symbols is empty, we can use even a simpler version of command.build_ext:

...
from distutils.command.build_ext import build_ext
def get_export_symbols_fixed(self, ext):
    pass  # return [] also does the job!

# replace wrong version with the fixed:
build_ext.get_export_symbols = get_export_symbols_fixed
...

This is even better than the first version, as it also fixes warning LNK4197!