python-multiprocessing

问题 I have been tackling this problem for a week now and it's been getting pretty frustrating because every time I implement a simpler but similar scale example of what I need to do, it turns out multiprocessing will fudge it up. The way it handles shared memory baffles me because it is so limited, it can become useless quite rapidly. So the basic description of my problem is that I need to create a process that gets passed in some parameters to open an image and create about 20K patches of size

问题 I was learning about multi-processing and multi-threading. From what I understand, threads run on the same core, so I was wondering if I create multiple processes inside a child thread will they be limited to that single core too? I'm using python, so this is a question about that specific language but I would like to know if it is the same thing with other languages? 回答1: I'm not a pyhton expert but I expect this is like in other languages, because it's an OS feature in general. Process A

问题 I've found this article which explains how to kill running multiprocessing code using ctr+c. Following code is fully working (it can be terminated it using ctrl+c): #!/usr/bin/env python # Copyright (c) 2011 John Reese # Licensed under the MIT License import multiprocessing import os import signal import time def init_worker(): signal.signal(signal.SIGINT, signal.SIG_IGN) def run_worker(): time.sleep(15) def main(): print "Initializng 5 workers" pool = multiprocessing.Pool(5, init_worker)

问题 Following code not executing as expected. import multiprocessing lock = multiprocessing.Lock() def dummy(): def log_results_l1(results): lock.acquire() print("Writing results", results) lock.release() def mp_execute_instance_l1(cmd): print(cmd) return cmd cmds = [x for x in range(10)] pool = multiprocessing.Pool(processes=8) for c in cmds: pool.apply_async(mp_execute_instance_l1, args=(c, ), callback=log_results_l1) pool.close() pool.join() print("done") dummy() But it does work if the

问题 This is my first venture into parallel processing and I have been looking into Dask but I am having trouble actually coding it. I have had a look at their examples and documentation and I think dask.delayed will work best. I attempted to wrap my functions with the delayed(function_name), or add an @delayed decorator, but I can't seem to get it working properly. I preferred Dask over other methods since it is made in python and for its (supposed) simplicity. I know dask doesn't work on the for

问题 The code below places three numbers in a queue. Then it attempts to get the numbers back from the queue. But it never does. How to get the data from the queue? import multiprocessing queue = multiprocessing.Queue() for i in range(3): queue.put(i) while not queue.empty(): print queue.get() 回答1: I originally deleted this answer after I read @Martijn Pieters', since he decribed the "why this doesn't work" in more detail and earlier. Then I realized, that the use case in OP's example doesn't

问题 I'm using Keras with Tensorflow as backend. I am trying to save a model in my main process and then load/run (i.e. call model.predict ) within another process. I'm currently just trying the naive approach from the docs to save/load the model: https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model. So basically: model.save() in main process model = load_model() in child process model.predict() in child process However, it simply hangs on the load_model call. Searching around I've

问题 I am trying to to make use of multiprocessing across several different computers, which pathos seems geared towards: "Pathos is a framework for heterogenous computing. It primarily provides the communication mechanisms for configuring and launching parallel computations across heterogenous resources." In looking at the documentation, however, I am at a loss as to how to get a cluster up and running. I am looking to: Set up a remote server or set of remote servers with secure authentication.

问题 Everywhere I see shared memory implementations for python (e.g. in multiprocessing ), creating shared memory always allocates new memory. Is there a way to create a shared memory object and have it refer to existing memory? The purpose would be to pre-initialize the data values, or rather, to avoid having to copy into the new shared memory if we already have, say, an array in hand. In my experience, allocating a large shared array is much faster than copying values into it. 回答1: The short

问题 I want to use an array for shared memory. The problem is the program is structured in such a way that the child processes are spawned before I know the size of the shared array. If I send a message to extend the array nothing happens and if I try to send the shared array itself I get an error. Below is a small script to demonstrate my problem. import multiprocessing as mp import numpy as np def f(a,pipe): while True: message, data = pipe.recv() if message == 'extend': a = np.zeros(data) print