python-multiprocessing

I am trying to learn multiprocessing with python. I wrote a simple code that should feed each process with 1000 lines from a txt input file. My main function reads a line, splits it and then performs some very simple operations with the elements in the string. Eventually the results should be written in an output file. When I run it, 4 processes are correctly spawned, but only one process is actually running with minimal CPU. As a result the code is very slow and defies the purpose to use multiprocessing in the first place. I think I don't have a global list problem like in this question (

In the code below, how do I make the Starter object be able to read gen.vals ? It seems like a different object gets created, whose state gets updated, but Starter never knows about it. Also, how would the solution apply for self.vals being a dictionary, or any other kind of object? import multiprocessing import time class Generator(multiprocessing.Process): def __init__(self): self.vals = [] super(Generator, self).__init__() def run(self): i = 0 while True: time.sleep(1) self.vals.append(i) print 'In Generator ', self.vals # prints growing list i += 1 class Starter(): def do_stuff(self): gen

I've followed Imanol Luengo 's answer to build a partial fit and transform for sklearn.decomposition.IncrementalPCA . But for some reason, it looks like (from htop) it uses all CPU cores at maximum. I could find neither n_jobs parameter nor anything related to multiprocessing. My question is: if this is default behavior of these functions how can I set the number of CPU's and where can I find information about it? If not, obviously I am doing something wrong in previous sections of my code. PS: I need to limit the number of CPU cores because using all cores in a server causing a lot of trouble

I am using Python's multiprocessing to create a parallel application. Processes need to share some data, for which I use a Manager . However, I have some common functions which processes need to call and which need to access the data stored by the Manager object. My question is whether I can avoid needing to pass the Manager instance to these common functions as an argument and rather use it like a global. In other words, consider the following code: import multiprocessing as mp manager = mp.Manager() global_dict = manager.dict(a=[0]) def add(): global_dict['a'] += [global_dict['a'][-1]+1] def