Using 100% of all cores with the multiprocessing module

Question

I have two pieces of code that I\'m using to learn about multiprocessing in Python 3.1. My goal is to use 100% of all the available processors. However, the code snippets he

Answer 1

Regarding code snippet 1: How many cores / processors do you have on your test machine? It isn't doing you any good to run 50 of these processes if you only have 2 CPU cores. In fact you're forcing the OS to spend more time context switching to move processes on and off the CPU than do actual work.

Try reducing the number of spawned processes to the number of cores. So "for i in range(50):" should become something like:

import os;
# assuming you're on windows:
for i in range(int(os.environ["NUMBER_OF_PROCESSORS"])):
    ...

Regarding code snippet 2: You're using a multiprocessing.Lock which can only be held by a single process at a time so you're completely limiting all the parallelism in this version of the program. You've serialized things so that process 1 through 50 start, a random process (say process 7) acquires the lock. Processes 1-6, and 8-50 all sit on the line:

l.acquire()

While they sit there they are just waiting for the lock to be released. Depending on the implementation of the Lock primitive they are probably not using any CPU, they're just sitting there using system resources like RAM but are doing no useful work with the CPU. Process 7 counts and prints to 1000 and then releases the lock. The OS then is free to schedule randomly one of the remaining 49 processes to run. Whichever one it wakes up first will acquire the lock next and run while the remaining 48 wait on the Lock. This'll continue for the whole program.

Basically, code snippet 2 is an example of what makes concurrency hard. You have to manage access by lots of processes or threads to some shared resource. In this particular case there really is no reason that these processes need to wait on each other though.

So of these two, Snippet 1 is closer to more efficiently utilitizing the CPU. I think properly tuning the number of processes to match the number of cores will yield a much improved result.