pool

Here is a complete simple working example import multiprocessing as mp import time import random class Foo: def __init__(self): # some expensive set up function in the real code self.x = 2 print('initializing') def run(self, y): time.sleep(random.random() / 10.) return self.x + y def f(y): foo = Foo() return foo.run(y) def main(): pool = mp.Pool(4) for result in pool.map(f, range(10)): print(result) pool.close() pool.join() if __name__ == '__main__': main() How can I modify it so Foo is only initialized once by each worker, not every task? Basically I want the init called 4 times, not 10. I am

I'm in the middle of a problem where I am unable decide which solution to take. The problem is a bit unique. Lets put it this way, i am receiving data from the network continuously (2 to 4 times per second). Now each data belongs to a different, lets say, group. Now, lets call these groups, group1, group2 and so on. Each group has a dedicated job queue where data from the network is filtered and added to its corresponding group for processing. At first I created a dedicated thread per group which would take data from the job queue, process it and then goes to blocking state (using Linked

So I've been messing around with python's multiprocessing lib for the last few days and I really like the processing pool. It's easy to implement and I can visualize a lot of uses. I've done a couple of projects I've heard about before to familiarize myself with it and recently finished a program that brute forces games of hangman. Anywho, I was doing an execution time compairison of summing all the prime numbers between 1 million and 2 million both single threaded and through a processing pool. Now, for the hangman cruncher, putting the games in a processing pool improved execution time by

Most examples of the Multiprocess Worker Pools execute a single function in different processes, f.e. def foo(args): pass if __name__ == '__main__': pool = multiprocessing.Pool(processes=30) res=pool.map_async(foo,args) Is there a way to handle two different and independent functions within the pool? So that you could assign f.e. 15 processes for foo() and 15 processes for bar() or is a pool bounded to a single function? Or du you have to create different processes for different functions manually with p = Process(target=foo, args=(whatever,)) q = Process(target=bar, args=(whatever,)) q.start(

For my college project I am trying to develop a python based traffic generator.I have created 2 CentOS machines on vmware and I am using 1 as my client and 1 as my server machine. I have used IP aliasing technique to increase number of clients and severs using just single client/server machine. Upto now I have created 50 IP alias on my client machine and 10 IP alias on my server machine. I am also using multiprocessing module to generate traffic concurrently from all 50 clients to all 10 servers. I have also developed few profiles(1kb,10kb,50kb,100kb,500kb,1mb) on my server(in /var/www/html

I try to organize pool with maximum 10 concurrent downloads. The function should download base url, then parser all urls on this page and download each of them, but OVERALL number of concurrent downloads should not exceed 10. from lxml import etree import gevent from gevent import monkey, pool import requests monkey.patch_all() urls = [ 'http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org', 'http://stackoverflow.com', # ... another 100 urls ] LINKS_ON_PAGE=[] POOL = pool.Pool(10) def parse_urls(page): html = etree.HTML(page) if html: links = [link for link in html.xpath("//a