python-multiprocessing

问题 I'm running a backup script that launches child processes to perform backups by rsync. However I have no way to limit the number of rsyncs it launches at a time. Here's the code I'm working on at the moment: print "active_children: ", multiprocessing.active_children() print "active_children len: ", len(multiprocessing.active_children()) while len(multiprocessing.active_children()) > 49: sleep(2) p = multiprocessing.Process(target=do_backup, args=(shash["NAME"],ip,shash["buTYPE"], )) jobs

问题 I'm trying to run some sample code of the multiprocessing.pool module of python, found in the web. The code is: def square(x): return x * x if __name__ == '__main__': pool = Pool(processes=4) inputs = [0, 1, 2, 3, 4] outputs = pool.map(square, inputs) But when i try to run it, it never finsh the execution and i have to restart the kernel of my IpythonNotebook notebook. What's the problem? 回答1: As you may read from the answer pointed out by John in the comments, multiprocessing.Pool , in

问题 This is pretty much the same as this question, but the given solution there (calling freeze_support()) does not work for me. I have the following script called start.py that I use to build a standalone executable with py2exe (version 0.9.2.2). I also have python.exe in the same directory. import multiprocessing def main(): print('Parent') p = multiprocessing.Process(target=new_process) multiprocessing.set_executable('python.exe') p.start() p.join() def new_process(): print('Child') if __name_

问题 I have a generator function (Python) that works like this def Mygenerator(x, y, z, ...): while True: # code that makes two matrices based on sequences of input arrays yield (matrix1, matrix2) What I want to do is to add the output from this generator. This line does the job: M1, M2 = reduce(lambda x, y: x[0] + y[0], x[1] + y[1], Mygenerator(x, y, z, ...)) I would like to parallelize this to speed up the computations. It is important that the outputs from Mygenerator is reduced as it is

问题 util.py def exec_multiprocessing(self, method, args): with concurrent.futures.ProcessPoolExecutor() as executor: results = pool.map(method, args) return results clone.py def clone_vm(self, name, first_run, host, ip): # clone stuff invoke.py exec_args = [(name, first_run, host, ip) for host, ip in zip(hosts, ips)] results = self.util.exec_multiprocessing(self.clone.clone_vm, exec_args) The above code gives the pickling error. I found that it is because we are passing instance method. So we

问题 I am using the multiprocessing functions of Python to run my code parallel on a machine with roughly 500GB of RAM. To share some arrays between the different workers I am creating a Array object: N = 150 ndata = 10000 sigma = 3 ddim = 3 shared_data_base = multiprocessing.Array(ctypes.c_double, ndata*N*N*ddim*sigma*sigma) shared_data = np.ctypeslib.as_array(shared_data_base.get_obj()) shared_data = shared_data.reshape(-1, N, N, ddim*sigma*sigma) This is working perfectly for sigma=1 , but for

问题 I have a set of images in a folder that I want to preprocess using some OpenCV functions. The function detectAndaligncrop takes an image path preprocesses it using OpenCV and returns the utput image. I am able to do it using: for image_path in files_list: cropped_image, _=detectAndaligncrop(im) cv2.imwrite("ouput_folder/{}".format(os.path.basename(image_path)),cropped_im*255.) However this is not working: jobs=[] for im_no, im in enumerate(files_list): p=multiprocessing.Process(target=saveIm

问题 I'm struggling again to improve the execution time of this piece of code. Since the calculations are really time-consuming I think that the best solution would be to parallelize the code. I was first working with maps as explained in this question, but then I tried a more simple approach thinking that I could find a better solution. However I couldn't come up with anything yet, so since it's a different problem I decided to post it as a new question. I am working on a Windows platform, using

问题 We are running a script using the multiprocessing library ( python 3.6 ), where a big pd.DataFrames is passed as an argument to a function : from multiprocessing import Pool import time def my_function(big_df): # do something time consuming time.sleep(50) if __name__ == '__main__': with Pool(10) as p: res = {} output = {} for id, big_df in some_dict_of_big_dfs: res[id] = p.apply_async(my_function,(big_df ,)) output = {u : res[id].get() for id in id_list} The problem is that we are getting an

问题 I use multiprocessing.connection.Listener for communication between processes, and it works as a charm for me. Now i would really love my mainloop to do something else between commands from client. Unfortunately listener.accept() blocks execution until connection from client process is established. Is there a simple way of managing non blocking check for multiprocessing.connection? Timeout? Or shall i use a dedicated thread? # Simplified code: from multiprocessing.connection import Listener