Can Cython code be compiled to a dll so C++ application can call it?

Question

I have a C++ program and it has sort of plugin structure: when program starts up, it\'s looking for dll in the plugin folder with certain exported function signatures, such

Answer 1

Using cython-module in a dll is not unlike using a cython-module in an embeded python interpreter.

The first step would be to mark cdef-function which should be used from external C-code with public, for example:

#cyfun.pyx:

#doesn't need python interpreter
cdef public int double_me(int me):
    return 2*me;

#needs initialized python interpreter
cdef public void print_me(int me):
    print("I'm", me);

cyfun.c and cyfun.h can be generated with

cython -3 cyfun.pyx

These files will be used for building of the dll.

The dll will need one function to initialize the python interpreter and another to finalize it, which should be called only once before double_me and print_me can be used (Ok, double_me would work also without interpreter, but this is an implementation detail).

The header-file for the dll would look like following:

//cyfun_dll.h
#ifdef BUILDING_DLL
    #define DLL_PUBLIC __declspec(dllexport) 
#else
    #define DLL_PUBLIC __declspec(dllimport) 
#endif

//return 0 if everything ok
DLL_PUBLIC int cyfun_init();
DLL_PUBLIC void cyfun_finalize();

DLL_PUBLIC int cyfun_double_me(int me);
DLL_PUBLIC void cyfun_print_me(int me);

So there are the necessary init/finalize-functions and the symbols are exported via DLL_PUBLIC (which needs to be done see this SO-post) so it can be used outside of the dll.

The implementation follows in cyfun_dll.c-file:

//cyfun_dll.c
#define BUILDING_DLL
#include "cyfun_dll.h"

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "cyfun.h"

DLL_PUBLIC int cyfun_init(){
  int status=PyImport_AppendInittab("cyfun", PyInit_cyfun);
  if(status==-1){
    return -1;//error
  } 
  Py_Initialize();
  PyObject *module = PyImport_ImportModule("cyfun");

  if(module==NULL){
     Py_Finalize();
     return -1;//error
  }
  return 0;   
}


DLL_PUBLIC void cyfun_finalize(){
   Py_Finalize();
}

DLL_PUBLIC int cyfun_double_me(int me){
    return double_me(me);
}

DLL_PUBLIC void cyfun_print_me(int me){
    print_me(me);
}

Noteworthy details:

1. we define BUILDING_DLL so DLL_PUBLIC becomes __declspec(dllexport).
2. we use cyfun.h generated by cython from cyfun.pyx.
3. cyfun_init inizializes python interpreter and imports the built-in module cyfun. The somewhat complicated code is because since Cython-0.29 PEP-489 is default. More information can be found in this SO-post.
   4. cyfun_double_me just wraps double_me so it becomes visible outside of the dll.

Now we can build the dll!

:: set up tool chain
call "<path_to_vcvarsall>\vcvarsall.bat" x64

:: build cyfun.c generated by cython
cl  /Tccyfun.c /Focyfun.obj /c <other_coptions> -I<path_to_python_include> 

:: build dll-wrapper
cl  /Tccyfun_dll.c /Focyfun_dll.obj /c <other_coptions> -I<path_to_python_include>

:: link both obj-files into a dll
link  cyfun.obj cyfun_dll.obj /OUT:cyfun.dll /IMPLIB:cyfun.lib /DLL <other_loptions> -L<path_to_python_dll>

The dll is now built, but the following details are noteworthy:

1. <other_coptions> and <other_loptions> can vary from installation to installation. An easy way is to see them is to runcythonize some_file.pyx` and to inspect the log.
2. we don't need to pass python-dll, because it will be linked automatically, but we need to set the right library-path.
3. we have the dependency on the python-dll, so later on it must be somewhere where it can be found.

---

Were you go from here depends on your task, we test our dll with a simple main:

//test.c
#include "cyfun_dll.h"

int main(){
   if(0!=cyfun_init()){
      return -1;
   }
   cyfun_print_me(cyfun_double_me(2));
   cyfun_finalize();
   return 0;
}

which can be build via

...
:: build main-program
cl  /Tctest.c /Focytest.obj /c <other_coptions> -I<path_to_python_include>

:: link the exe
link test.obj cyfun.lib /OUT:test_prog.exe <other_loptions> -L<path_to_python_dll>

And now calling test_prog.exe leads to the expected output "I'm 4".

Depending on your installation, following things must be considered:

• test_prog.exe depends on pythonX.Y.dll which should be somewhere in the path so it can be found (the easiest way is to copy it next to the exe)
• The embeded python interpreter needs an installation, see this and/or this SO-posts.

---

IIRC, it is not a great idea to initialize, then to finalize and then to initialize the Python-interpreter again (that might work for some scenarios, but not all , see for example this) - the interpreter should be initialized only once and stay alive until the programs ends.

So if your C/C++-program already has an initialized Python-interpreter it would make sense to offer a function which only imports the module cyfun and doesn't initialize. In this case I would define CYTHON_PEP489_MULTI_PHASE_INIT=0, because PyImport_AppendInittab must be called before Py_Initialize, which might be already too late when the dll is loaded.

Answer 2

I'd imagine it'd be difficult to call it directly, with Cython depending so much on the Python runtime.

Your best bet is to embed a Python interpreter directly inside your app, e.g. as described in this answer, and call your Cython code from the interpreter. That's what I would do.