Python: GIL context - switching

Question

So, I generally have a pretty good understanding of how the Global Interpreter Lock (GIL) in Python works. Essentially, while the interpreter is running, one thread holds the GIL for N ticks (where N can be set using sys.setcheckinterval), at which point the GIL is released and another thread can acquire the GIL. The also happens if one thread begins an I/O operation.

What I'm a bit confused about is how this all works with C extension modules.

If you have a C extension module that acquires the GIL, and then executes some python code using PyEval_EvalCode, can the interpreter release the GIL and give it to some other thread? Or will the C thread that acquired the GIL hold the GIL permanently until PyEval_EvalCode returns and the GIL is explicitly released in C?

PyGILState gstate = PyGILState_Ensure();

....

/* Can calling PyEval_EvalCode release the GIL and let another thread acquire it?? */
PyObject* obj = PyEval_EvalCode(code, global_dict, local_dict); 

PyGILState_Release(gstate);

Answer 1

Yes, the interpreter can always release the GIL; it will give it to some other thread after it has interpreted enough instructions, or automatically if it does some I/O. Note that since recent Python 3.x, the criteria is no longer based on the number of executed instructions, but on whether enough time has elapsed.

To get a different effect, you'd need a way to acquire the GIL in "atomic" mode, by asking the GIL not to be released until you release it explicitly. This is impossible so far (but see https://bitbucket.org/arigo/cpython-withatomic for an experimental version).

Answer 2

As Armin said, the GIL can be released inside PyEval_EvalCode. When it returns, it is of course acquired again.

The best way is just to make sure that your code can handle that. For example, incref any objects where you have C pointers to before the GIL might get released. Also, be careful if there might be cases where the Python code again calls the very same function. If you have another mutex there, you can easily end up in a dead-lock. Use recursive-safe mutexes and while waiting on them, you should release the GIL so that the original thread can releases such mutexes.