python-multiprocessing

I try to utilize the pool multiprocessing functionality of python. Independent how I set the chunk size (under Windows 7 and Ubuntu - the latter see below with 4 cores), the amount of parallel threads seems to stay the same. from multiprocessing import Pool from multiprocessing import cpu_count import multiprocessing import time def f(x): print("ready to sleep", x, multiprocessing.current_process()) time.sleep(20) print("slept with:", x, multiprocessing.current_process()) if __name__ == '__main__': processes = cpu_count() print('-' * 20) print('Utilizing %d cores' % processes) print('-' * 20)

I'm doing some multiprocessing python scripts using multiprocessing.Pool . These scripts look like the following: from multiprocessing import Pool def f(x): return x*x if __name__ == '__main__': with Pool(processes=4) as pool: # start 4 worker processes print(pool.map(f, range(10))) # prints "[0, 1, 4,..., 81]" When running this with Python 3.4, everything is fine. However, when using Python 2.6 or 3.1 I get this error: AttributeError: 'Pool' object has no attribute '__exit__' Using Python 2.7 or 3.2 , the error is essentially the same: AttributeError: __exit__ Why does this happen and how can

I have a dict to store objects: jobs = {} job = Job() jobs[job.name] = job now I want to convert it to use manager dict because I want to use multiprocessing and need to share this dict amonst processes mgr = multiprocessing.Manager() jobs = mgr.dict() job = Job() jobs[job.name] = job just by converting to use manager.dict() things got extremely slow. For example, if using native dict, it only took .65 seconds to create 625 objects and store it into the dict. The very same task now takes 126 seconds! Any optimization i can do to keep manager.dict() on par with python {}? The problem is that

I have a multiprocessing program where one process adds elements to a shared list ( multiprocessing.Manager().list() ) several other processes consume these elements from that list (and remove them); they run until there is something to process in the list and the process above is still adding to the list. I implemented locking (via multiprocessing.Lock() ) when adding to the list, or removing from it. Since there is one "feeder" process and several (10-40) "consumer" ones all competing for the lock, and that the consumer processes are fast, I end up with the "feeder" process having a hard

I have a multi processed web server with processes that never end, I would like to check my code coverage on the whole project in a live environment (not only from tests). The problem is, that since the processes never end, I don't have a good place to set the cov.start() cov.stop() cov.save() hooks. Therefore, I thought about spawning a thread that in an infinite loop will save and combine the coverage data and then sleep some time, however this approach doesn't work, the coverage report seems to be empty, except from the sleep line. I would be happy to receive any ideas about how to get the