If the one and only Python interpreter is in the middle of executing a bytecode when the OS dispatches another thread, which calls a Python callback - what happens? Am I right to be concerned about the reliability of this design?
6
votes
Are you talking about the GIL? See this pycon video on how the GIL works
- Joel Cornett
I'm analyzing an existing C library. Neither threading nor multiprocessing is used on the Python side. That's what is causing me some concerns.
- phunctor
I have a merely mortal understanding of the GIL. I'm assuming there are excellent reasons that it is managed by threading. The absence of that management here worries me.
- phunctor
1 Answers
3
votes
In general situations, the C library needs to call PyEval_InitThreads() to gain GIL before spawning any thread that invokes python callbacks. And the callbacks need to be surrounded with PyGILState_Ensure() and PyGILState_Release() to ensure safe execution.
However, if the C library is running within the context of, say, a python C extension, then there are simple cases where it is safe to omit GIL manipulation at all.
Think of this calling sequence: 1) python code calls C function foo(), 2) foo() spawns one and only one thread that runs another C function bar(), which calls back to python code, and 3) foo() always joins or cancels the thread running bar() before returns.
In such a case, it is safe to omit GIL manipulation. Because foo() owns the GIL (i.e. implicitly borrowed from the python code that calls it) during its lifecycle, and the execution of python callbacks within the lifecycle of foo() is serialized (i.e. only one callback thread and python code does not incorporate threading).
