import inside of a Python thread

Question

I have some functions that interactively load python modules using __import__

I recently stumbled upon some article about an "import lock" in Python, that is, a lock specifically for imports (not just the GIL). But the article was old so maybe that's not true anymore.

This makes me wonder about the practice of importing in a thread.

1. Are import/__import__ thread safe?
2. Can they create dead locks?
3. Can they cause performance issues in a threaded application?

EDIT 12 Sept 2012

Thanks for the great reply Soravux. So import are thread safe, and I'm not worrying about deadlocks, since the functions that use __import__ in my code don't call each others.

Do you know if the lock is acquired even if the module has already been imported ? If that is the case, I should probably look in sys.modules to check if the module has already been imported before making a call to __import__.

Sure this shouldn't make a lot of difference in CPython since there is the GIL anyway. However it could make a lot of difference on other implementations like Jython or stackless python.

EDIT 19 Sept 2012

About Jython, here's what they say in the doc:

http://www.jython.org/jythonbook/en/1.0/Concurrency.html#module-import-lock

Python does, however, define a module import lock, which is implemented by Jython. This lock is acquired whenever an import of any name is made. This is true whether the import goes through the import statement, the equivalent __import__ builtin, or related code. It’s important to note that even if the corresponding module has already been imported, the module import lock will still be acquired, if only briefly.

So, it seems that it would make sense to check in sys.modules before making an import, to avoid acquiring the lock. What do you think?

Answer 1

Normal imports are thread safe because they acquire an import lock prior to execution and release it once the import is done. If you add your own custom imports using the hooks available, be sure to add this locking scheme to it. Locking facilities in Python may be accessed by the imp module (imp.lock_held()/acquire_lock()/release_lock()).

Using this import lock won't create any deadlocks or dependency errors aside from the circular dependencies that are already known.

The low-level call to create a thread being clone on Linux, threading in Python is then a fork-like operation. Forking and cloning applies different behaviors on the various memory segments. For exemple, only the stack is not shared by threads, compared to forks which clones more segments (Data (often COW), Stack, Code, Heap), effectively not sharing its content. The import mechanism in Python uses the global namespace which is not placed on the stack, thus using a shared segment with its threads. Since the side-effects (ie. the changes in memory) of importing works in the same segments, it behaves as a single-threaded program. Be careful to use thread safe libraries in your imports on multithreaded programs, though. It will cause mayhem to use calls to functions that are not thread safe in such environment.

By the way, threaded programs in Python suffers the GIL which won't allow much performance gains unless your program is I/O bound or rely on C or external thread-safe libraries (since they release the GIL before executing). Running in two threads the same imported function won't execute concurrently because of this GIL. Note that this is only a limitation of CPython and other implementations of Python will have a different behavior.

To answer your edit: imported modules are all cached by Python. If the module is already loaded in the cache, it won't be run again and the import statement (or function) will return right away. You don't have to implement yourself the cache lookup in sys.modules, Python does that for you and won't imp lock anything, aside from the GIL for the sys.modules lookup.

To answer your second edit: I prefer having to maintain a simpler code than trying to optimize calls to the libraries I use (in this case, the standard library). The rationale is that the time required to perform something is usually way more important than the time required to import the module that does it. Furthermore, the time required to maintain this kind of code throughout the project is way higher than the time it will take to execute. It all boils down to: "programmer time is more valuable than CPU time".