Multithreading with Python and C api

Question

I have a C++ program that uses the C api to use a Python library of mine. Both the Python library AND the C++ code are multithreaded.

In particular, one th

Answer 1

A correct order of steps to perform what you are trying to do is:

• In the main thread:

  1. Initialize Python using Py_Initialize*.
  2. Initialize Python threading support using PyEval_InitThreads().
  3. Start the C++ thread.

At this point, the main thread still holds the GIL.

• In a C++ thread:
  1. Acquire the GIL using PyGILState_Ensure().
  2. Create a new Python thread object and start it.
  3. Release the GIL using PyGILState_Release().
  4. Sleep, do something useful or exit the thread.

Because the main thread holds the GIL, this thread will be waiting to acquire the GIL. If the main thread calls the Python API it may release the GIL from time to time allowing the Python thread to execute for a little while.

• Back in the main thread:
  1. Release the GIL, enabling threads to run using PyEval_SaveThread()
  2. Before attempting to use other Python calls, reacquire the GIL using PyEval_RestoreThread()

I suspect that you are missing the last step - releasing the GIL in the main thread, allowing the Python thread to execute.

I have a small but complete example that does exactly that at this link.

Answer 2

You probably do not unlock the Global Interpreter Lock when you callback from python's threading.Thread.

Well, if you are using bare python's C API you have some documentation here, about how to release/acquire the GIL. But while using C++, I must warn you that it might broke down upon any exceptions throwing in your C++ code. See here.

In general any of your C++ function that runs for too long should unlock GIL and lock, whenever it use the C Python API again.