Python PyGILState_{Ensure/Release} causes segfault while returning to C++ from Python code

Question

UPDATE Well, it looks like adding PyEval_InitThreads() before the call to PyGILState_Ensure() does the trick. In my haste to figure things out I incorrectly

Answer 1

I ran into this exact problem as well. The documentation for anything relating to threads in CPython is unfortunately patchy at best.

Essentially, you need to do the following:

In your main thread, BEFORE any other threads are spawned, you need to call PyEval_InitThreads(). A good place to do this is right after you call PyInitialize().

Now, PyEval_InitThreads() not only initializes the Python interpreter thread-state, it also implicitly acquires the Global Interpreter Lock. This means, you need to release the lock before you call PyGILEnsure_State() in some other thread, otherwise your program will hang. You can do this with the function PyEval_ReleaseLock().

So basically, in your main thread, before any other threads are launched, you want to say:

PyInitialize();
PyEval_InitThreads();
PyEval_ReleaseLock();

Then, in any additional thread, anytime you use the Python API you need to say:

PyGILState_STATE gstate;
gstate = PyGILState_Ensure();

/* ... some code that does things with Python ... */

PyGILState_Release(gstate); 

Answer 2

Python expects a certain amount of initialization to be done by the main thread before anything attempts to call back in from a subthread.

If the main thread is an application that is embedding Python, then it should call PyEval_InitThreads() immediately after calling Py_Initialize().

If the main thread is instead the Python interpreter itself (as seems to be the case here), then the module using the multithreaded extension module should include an "import threading" early to ensure that PyEval_InitThreads() is called correctly before any subthreads are spawned.