python-multiprocessing

问题 I am using Python's multiprocessing to create a parallel application. Processes need to share some data, for which I use a Manager . However, I have some common functions which processes need to call and which need to access the data stored by the Manager object. My question is whether I can avoid needing to pass the Manager instance to these common functions as an argument and rather use it like a global. In other words, consider the following code: import multiprocessing as mp manager = mp

问题 I am using Python's multiprocessing to create a parallel application. Processes need to share some data, for which I use a Manager . However, I have some common functions which processes need to call and which need to access the data stored by the Manager object. My question is whether I can avoid needing to pass the Manager instance to these common functions as an argument and rather use it like a global. In other words, consider the following code: import multiprocessing as mp manager = mp

问题 I have a (large) list with male and female agentes. I want to apply different functions to each. How can I use Pool in such a case? Given that the agents are independent of each other. An example would be: males = ['a', 'b', 'c'] females = ['d', 'e', 'f'] for m in males: func_m(m) for f in females: func_f(f) I started like that: from multiprocessing import Pool p = Pool(processes=2) p.map() # Here is the problem I would like to have something like: p.ZIP(func_f for f in females, func_m for m

问题 When I run predict_proba on a dataframe without multiprocessing I get the expected behavior. The code is as follows: probabilities_data = classname.perform_model_prob_predictions_nc(prediction_model, vectorized_data) where: perform_model_prob_predictions_nc is: def perform_model_prob_predictions_nc(model, dataFrame): try: return model.predict_proba(dataFrame) except AttributeError: logging.error("AttributeError occurred",exc_info=True) But when I try to run the same function using chunks and

问题 I am running this code from multiprocessing.Pool import ThreadPool from selenium import webdriver driver = webdriver.Firefox() driver.get(url) elements = driver.find_elements_by_class_name("class-name") pool = ThreadPool(4) async = [pool.apply_async(fn_which_calls_get_attribute,(element,)) for element in elements] results = [result.get() for result in async] which works fine for some of the results, but throws an error of ResponseNotReady for other results. It runs as expected if I use "pool

问题 I have a simple python multiprocessing script that sets up a pool of workers that attempt to append work-output to a Manager list. The script has 3 call stacks: - main calls f1 that spawns several worker processes that call another function g1. When one attempts to debug the script (incidentally on Windows 7/64 bit/VS 2010/PyTools) the script runs into a nested process creation loop, spawning an endless number of processes. Can anyone determine why? I'm sure I am missing something very simple

问题 In Python (2.7) I try to create processes (with multiprocessing) in a celery task (celery 3.1.17) but it gives the error: daemonic processes are not allowed to have children Googling it, I found that most recent versions of billiard fix the "bug" but I have the most recent version (3.3.0.20) and the error is still happening. I also tried to implement this workaround in my celery task but it gives the same error. Does anybody know how to do it? Any help is appreciated, Patrick EDIT: snippets

问题 I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores. I wrote the following code, which works if I use normal map , but with pool.map returns an error. import multiprocessing pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1) class OtherClass: def run(sentence, graph): return False class SomeClass: def __init__(self): self.sentences = [["Some string"]] self.graphs = ["string"] def some_method(self): other =

问题 I have a method inside a class that needs to do a lot of work in a loop, and I would like to spread the work over all of my cores. I wrote the following code, which works if I use normal map , but with pool.map returns an error. import multiprocessing pool = multiprocessing.Pool(multiprocessing.cpu_count() - 1) class OtherClass: def run(sentence, graph): return False class SomeClass: def __init__(self): self.sentences = [["Some string"]] self.graphs = ["string"] def some_method(self): other =

问题 For my college project I am trying to develop a python based traffic generator.I have created 2 CentOS machines on vmware and I am using 1 as my client and 1 as my server machine. I have used IP aliasing technique to increase number of clients and severs using just single client/server machine. Upto now I have created 50 IP alias on my client machine and 10 IP alias on my server machine. I am also using multiprocessing module to generate traffic concurrently from all 50 clients to all 10