Python Multiprocessing with a single function

Question

I have a simulation that is currently running, but the ETA is about 40 hours -- I\'m trying to speed it up with multi-processing.

It essentially iterates over 3 valu

Answer 1

• Wrap the data for each iteration up into a tuple.
• Make a list data of those tuples
• Write a function f to process one tuple and return one result
• Create p = multiprocessing.Pool() object.
• Call results = p.map(f, data)

This will run as many instances of f as your machine has cores in separate processes.

Edit1: Example:

from multiprocessing import Pool

data = [('bla', 1, 3, 7), ('spam', 12, 4, 8), ('eggs', 17, 1, 3)]

def f(t):
    name, a, b, c = t
    return (name, a + b + c)

p = Pool()
results = p.map(f, data)
print results

Edit2:

Multiprocessing should work fine on UNIX-like platforms such as OSX. Only platforms that lack os.fork (mainly MS Windows) need special attention. But even there it still works. See the multiprocessing documentation.