Python Global Interpreter Lock (GIL) workaround on multi-core systems using taskset on Linux?

Question

So I just finished watching this talk on the Python Global Interpreter Lock (GIL) http://blip.tv/file/2232410.

The gist of it is that the GIL is a pretty good design

Answer 1

This is a pretty old question but since everytime I search about information related to python and performance on multi-core systems this post is always on the result list, I would not let this past before me an do not share my thoughts.

You can use the multiprocessing module that rather than create threads for each task, it creates another process of cpython compier interpreting your code. It would make your application to take advantage of multicore systems. The only problem that I see on this approach is that you will have a considerable overhead by creating an entire new process stack on memory. (http://en.wikipedia.org/wiki/Thread_(computing)#How_threads_differ_from_processes)

Python Multiprocessing module: http://docs.python.org/dev/library/multiprocessing.html

"The reason I am not using the multiprocessing module is that (in this case) part of the program is heavily network I/O bound (HTTP requests) so having a pool of worker threads is a GREAT way to squeeze performance out of a box..."

About this, I guess that you can have also a pool of process too: http://docs.python.org/dev/library/multiprocessing.html#using-a-pool-of-workers

Att, Leo