python-multiprocessing

问题 I'm trying to implement metaclass/decorator facilities to allow easy-ish parallelization of code. I'm using Python's multiprocessing. Say I have: class Worker(metaclass=Parallelizable): def __init__(self): super().__init__() # annotate some method for parallele computation @ParalleleMethod def long_calculation(self, data): # do stuff return ans class ParalleleMethod: def __init__(self, func): self.func = func def __call__(self, data): # as a prototype of the idea I want to get to pool.starmap

问题 Im trying to understand multiprocessing.Process class. I want to collect data asynchronously storing it somewhere. After having stored the data, it somehow gets lost. Here is my MWE: from __future__ import print_function import multiprocessing as mp def append_test(tgt): tgt.append(42) print('Appended:', tgt) l = [] p = mp.Process(target=lambda: append_test(l)) p.run() print('l is', l) p.start() p.join() print('l is', l) If I'm running that snippet, I get Appended: [42] l is [42] Appended:

问题 I am trying a write a python code having multiple processes whose structure and flow is something like this: import multiprocessing import ctypes import time import errno m=multiprocessing.Manager() mylist=m.list() var1=m.Value('i',0) var2=m.Value('i',1) var3=m.Value('i',2) var4=m.Value(ctypes.c_char_p,"a") var5=m.Value(ctypes.c_char_p,"b") var6=3 var7=4 var8=5 var9=6 var10=7 def func(var1,var2,var4,var5,mylist): i=0 try: if var1.value==0: print var2.value,var4.value,var5.value mylist.append

问题 I am new to multiprocessing concepts in python and I have problem accessing variables when I try to include multiprocessing in my code. Sorry if Iam sounding naive, but I just cant figure it out. Below is a simple version of my scenario. class Data: def __init__(self): self.data = "data" def datameth(self): print self.data print mainvar class First: def __init__(self): self.first = "first" def firstmeth(self): d = Data() d.datameth() print self.first def mymethod(): f = First() f.firstmeth()

问题 Based on this answer, I'd like to run this multiprocessing job with Python 2.7 / Windows: def main(): import itertools as it from multiprocessing import Pool def dothejob(i, j, k): print i, j, k the_args = it.product(range(100), range(100), range(100)) pool = Pool(4) def jobWrapper(args): return dothejob(*args) res = pool.map(jobWrapper, the_args) if __name__ == '__main__': main() The main() and the last two lines are necessary because without them, there's the well known bug: This probably

问题 Inspired by this solution I am trying to set up a multiprocessing pool of worker processes in Python. The idea is to pass some data to the worker processes before they actually start their work and reuse it eventually. It's intended to minimize the amount of data which needs to be packed/unpacked for every call into a worker process (i.e. reducing inter-process communication overhead). My MCVE looks like this: import multiprocessing as mp import numpy as np def create_worker_context(): global

问题 Consider the following example: from multiprocessing import Queue, Pool def work(*args): print('work') return 0 if __name__ == '__main__': queue = Queue() pool = Pool(1) result = pool.apply_async(work, args=(queue,)) print(result.get()) This raises the following RuntimeError : Traceback (most recent call last): File "/tmp/test.py", line 11, in <module> print(result.get()) [...] RuntimeError: Queue objects should only be shared between processes through inheritance But interestingly the

问题 I am making a project which collects data from clients sensors, processes the gathered data and sends it back to clients. There can be multiple clients asking to receive some data from our server at the same time, so I had to implement multiprocessing. I can't use Threads because there are certain variables that must be client independent. If I did, my code would probably get very complicated to read and upgrade, and I don't want that. So I decided to use Processes, but now there is some data

问题 I'm trying to parallelize a function in my program for time measurement purposes, but I get an error and I don't know how to fix it. Here's the code: def evolucionAutomata(automata, regla, numero_evoluciones): if(numero_evoluciones == 0): return 0 with Pool(4) as p: automataEvolucionado = list(p.map(obtenerVecindario, automata, rotarDerecha(automata, 1), rotarIzquierda(automata, 1), lista_regla))** print(automataEvolucionado) evolucionAutomata(automataEvolucionado, regla, numero_evoluciones -

问题 I have a multiprocessing.Manager object that contains a multiprocessing.Queue to manage all of the work that I want a group of processes to do. I would like to get the number of elements left in this queue and was wondering how to do this? The len function does not work. 回答1: If the queue you are talking about is multiprocessing.Queue , try to use qsize() method for multiprocessing.Queue objects, but be careful: qsize() Return the approximate size of the queue. Because of multithreading