Python multiprocessing: restrict number of cores used

Question

I want to know how to distribute N independent tasks to exactly M processors on a machine that has L cores, where L>M. I don\'t want to use all the processors because I sti

Answer 1

You might want to look into the following package:

http://pypi.python.org/pypi/affinity

It is a package that uses sched_setaffinity and sched _getaffinity.

The drawback is that it is highly Linux-specific.

Answer 2

you might try using pypar module. I am not sure how to use affinity to set cpu affinity of to a certain core using affinity

Answer 3

If you are on linux, use taskset when you launch the program

A child created via fork(2) inherits its parent’s CPU affinity mask. The affinity mask is preserved across an execve(2).

TASKSET(1)
Linux User’s Manual
TASKSET(1)

NAME taskset - retrieve or set a process’s CPU affinity

SYNOPSIS taskset [options] mask command [arg]... taskset [options] -p [mask] pid

DESCRIPTION taskset is used to set or retrieve the CPU affinity of a running process given its PID or to launch a new COMMAND with a given CPU affinity. CPU affinity is a scheduler property that "bonds" a process to a given set of CPUs on the system. The Linux scheduler will honor the given CPU affinity and the process will not run on any other CPUs. Note that the Linux scheduler also supports natural CPU affinity: the scheduler attempts to keep processes on the same CPU as long as practical for performance reasons. Therefore, forcing a specific CPU affinity is useful only in certain applications.

The CPU affinity is represented as a bitmask, with the lowest order bit corresponding to the first logical CPU and the highest order bit corresponding to the last logical CPU. Not all CPUs may exist on a given sys‐ tem but a mask may specify more CPUs than are present. A retrieved mask will reflect only the bits that cor‐ respond to CPUs physically on the system. If an invalid mask is given (i.e., one that corresponds to no valid CPUs on the current system) an error is returned. The masks are typically given in hexadecimal.

Answer 4

Probably a dumb observation, pls forgive my inexperience in Python.

But your while loop polling for the finished tasks is not going to sleep and is consuming one core all time, isn't it?

The other thing to notice is that if your tasks are I/O bound, you M should be adjusted to the number of parallel disks(?) you have ... if they are NFS mounted in different machine you could potentially have M>L.

g'luck!

Answer 5

On my dual-core machine the total number of processes is honoured, i.e. if I do

p = Pool(1)

Then I only see one CPU in use at any given time. The process is free to migrate to a different processor, but then the other processor is idle. I don't see how all your processors can be in use at the same time, so I don't follow how this can be related to your I/O issues. Of course, if your simulation is I/O bound, then you will see sluggish I/O regardless of core usage...