Multithreading with Python and C api

Question

I have a C++ program that uses the C api to use a Python library of mine. Both the Python library AND the C++ code are multithreaded.

In particular, one thread of the C++ program instantiates a Python object that inherits from threading.Thread. I need all my C++ threads to be able to call methods on that object.

From my very first tries (I naively just instantiate the object from the main thread, then wait some time, then call the method) I noticed that the execution of the Python thread associated with the object just created stops as soon as the execution comes back to the C++ program.

If the execution stays with Python (for example, if I call PyRun_SimpleString("time.sleep(5)");) the execution of the Python thread continues in background and everything works fine until the wait ends and the execution goes back to C++.

I am evidently doing something wrong. What should I do to make both my C++ and Python multithreaded and capable of working with each other nicely? I have no previous experience in the field so please don't assume anything!

Answer 1

A correct order of steps to perform what you are trying to do is:

• In the main thread:

  1. Initialize Python using Py_Initialize*.
  2. Initialize Python threading support using PyEval_InitThreads().
  3. Start the C++ thread.

At this point, the main thread still holds the GIL.

• In a C++ thread:
  1. Acquire the GIL using PyGILState_Ensure().
  2. Create a new Python thread object and start it.
  3. Release the GIL using PyGILState_Release().
  4. Sleep, do something useful or exit the thread.

Because the main thread holds the GIL, this thread will be waiting to acquire the GIL. If the main thread calls the Python API it may release the GIL from time to time allowing the Python thread to execute for a little while.

• Back in the main thread:
  1. Release the GIL, enabling threads to run using PyEval_SaveThread()
  2. Before attempting to use other Python calls, reacquire the GIL using PyEval_RestoreThread()

I suspect that you are missing the last step - releasing the GIL in the main thread, allowing the Python thread to execute.

I have a small but complete example that does exactly that at this link.

Answer 2

You probably do not unlock the Global Interpreter Lock when you callback from python's threading.Thread.

Well, if you are using bare python's C API you have some documentation here, about how to release/acquire the GIL. But while using C++, I must warn you that it might broke down upon any exceptions throwing in your C++ code. See here.

In general any of your C++ function that runs for too long should unlock GIL and lock, whenever it use the C Python API again.