python-multiprocessing

问题 The code: import multiprocessing print(f'num cpus {multiprocessing.cpu_count():d}') import sys; print(f'Python {sys.version} on {sys.platform}') def _process(m): print(m) #; return m raise ValueError(m) args_list = [[i] for i in range(1, 20)] if __name__ == '__main__': with multiprocessing.Pool(2) as p: print([r for r in p.starmap(_process, args_list)]) prints: num cpus 8 Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28) [Clang 6.0 (clang-600.0.57)] on darwin 1 7 4 10 13 16 19

问题 This question already has an answer here : Running multiple threads at the same time (1 answer) Closed 2 years ago . I want to use Python's multiprocessing module to start a new process which creates some other object and calls that objects loops_forever method. In my main class I have: import OtherService from multiprocessing import Process my_other_service = OtherService(address=ADDRESS) my_other_process = Process(target=my_other_service.loops_forever()) print("got here") my_other_process

问题 I improved my execution time by using multiprocessing but I am not sure whether the behavior of the PC is correct, it freezes the system until all processes are done. I am using Windows 7 and Python 2.7. Perhaps I am doing a mistake, here is what I did: def do_big_calculation(sub_list, b, c): # do some calculations here with the sub_list if __name__ == '__main__': list = [[1,2,3,4], [5,6,7,8], [9,10,11,12]] jobs = [] for sub_l in list : j = multiprocessing.Process(target=do_big_calculation,

问题 A managers.dict() allow to share a dictionary across process and perform thread-safe operation. In my case each a coordinator process create the shared dict with m elements and n worker processes read and write to/from a single dict key. Do managers.dict() have one single lock for the dict or m locks, one for every key in it? Is there an alternative way to share m elements to n workers, other than a shared dict, when the workers do not have to communicate with each other? Related python

问题 My eventual goal is to redirect the stdout from several subprocesses to some queues, and print those out somewhere (maybe in a little GUI). The first step is to subclass Queue into an object that behaves much like the stdout . But that is where I got stuck. Subclassing the multiprocessing Queue seems impossible in Python v3.5. # This is a Queue that behaves like stdout # Unfortunately, doesn't work in Python 3.5 :-( class StdoutQueue(Queue): def __init__(self,*args,**kwargs): Queue.__init__

问题 I am creating, multiplying and then summing all elements of two big matrices in numpy. I do this some hundred times with two methods, a loop and with the help of the multiprocessing modul (see the snipet below). def worker_loop(n): for i in n: mul = np.sum(np.random.normal(size=[i,i])*np.random.normal(size=[i,i])) def worker(i): mul = np.sum(np.random.normal(size=[i,i])*np.random.normal(size=[i,i])) n = range(100,300) pool = ThreadPool(2) pool.map(worker, n) pool.close() pool.join() worker

问题 I have a process that is essentially just an infinite loop and I have a second process that is a timer. How can I kill the loop process once the timer is done? def action(): x = 0 while True: if x < 1000000: x = x + 1 else: x = 0 def timer(time): time.sleep(time) exit() loop_process = multiprocessing.Process(target=action) loop_process.start() timer_process = multiprocessing.Process(target=timer, args=(time,)) timer_process.start() I want the python script to end once the timer is done. 回答1:

问题 I have a list of files that I pass into a for loop and do a whole bunch of functions. Whats the easiest way to parallelize this? Not sure I could find this exact thing anywhere and I think my current implementation is incorrect because I only saw one file being run. From some reading I've done, I think this should be a perfectly parallel case. Old code is something like this: import pandas as pd filenames = ['file1.csv', 'file2.csv', 'file3.csv', 'file4.csv'] for file in filenames: file1 = pd

问题 from itertools import product f = open('filename.txt', 'a') def worker(i, j): print i,j f.write("%s\t%s\n"%(i,j)) return def main(): a_list = ['1', '2', '3', '4', '5'] #5 item b_list = ['6', '7', '8'] #3 item # Total 5*3=15 combinations from multiprocessing import Pool pool = Pool(processes=4) results = [pool.apply_async(worker, args=(i, j)) for i, j in product(a_list, b_list)] output = [p.get() for p in results] main() f.close() this is the code I'm trying to run and store result in a txt

问题 Python multiprocessing package supports a remote manager feature where one python process can IPC with another process, however from their example it seems this must go through the OS's IP stack. Is there a way of using the remote manager without going through the IP stack, assuming the two processes are local, thus making it quicker? 来源： https://stackoverflow.com/questions/8466624/remote-python-manager-without-going-through-ip-stack