python-multiprocessing

问题 I am trying to update a key in a nested dictionary of multiprocessing module's manager.dict() but not able to do so. It doesn't update the value and doesn't throw any error too. Code: import time import random from multiprocessing import Pool, Manager def spammer_task(d, token, repeat): success = 0 fail = 0 while success+fail<repeat: time.sleep(random.random()*2.0) if (random.random()*100)>98.0: fail+=1 else: success+=1 d[token] = { 'status': 'ongoing', 'fail': fail, 'success': success,

问题 I have this code that should put an event in a queue each time an external program (TCPdump) creates a *.pcap file in my directory. My problem is that I always get an empty queue, although I got the print from process() function. What am I doing wrong? Is the queue correctly defined and shared between the two classes? EDIT----------------- I maybe understood why I got an empty queue, I think it is because I'm printing the queue that I initialized before it gets filled by Handler class. I

问题 I am reading log files in my python code which contains some nested json data. I have a nested for loop containing 4 for-loops from which values of certain keys are extracted and appended to a dataframe. The nested for-loop is taking too much time and I saw from other answers that multiprocessing is the way to go for nested for-loops but did not find an example for json data. What is the best approach for this ? Below is my code to extract data from log files and into dataframes.

问题 I am working on task where I am creating multiple process to run code in parallel to speed up process. below is my code. def update_value(value): print('module name:\n', __name__) print('parent process:\n', os.getppid()) print('process id:\n', os.getpid()) value_read = server_connect_read(channel, value) if value_read.server_connect() is False: return False print("updating values") update = server_read.update_value(old_values.xlsx) if value_read.server_disconnet() is False: return False Pool

问题 I have a python code that performs filtering on a matrix. I have created a C++ interface using pybind11 that successfully runs in serialized fashion (please see the code in below). I am trying to make it parallel-processing to hopefully reduce the computation time compared to its serialized version. To do this, I have splitted my array of size M×N into three sub-matrices of size M×(N/3) to process them in parallel using the same interface. I used ppl.h library to make a parallel for-loop and

问题 I'm currently trying to use multiprocessing to process a big result obtained after a psycopg2 query. I split the result in several lists of 100 then send them for multiprocessing. When I do so though, I get the following error TypeError: can't pickle psycopg2.extensions.connection objects Here is my psycopg2 query: def get_employees(self): logging.info('POSTGRESQL QUERY: get_employees') try: self.cur = self.conn.cursor(cursor_factory=RealDictCursor) self.cur.execute( "SELECT ..." ) employees

问题 I want to use multiprocessing.Pool, but multiprocessing.Pool can't abort a task after a timeout. I found solution and some modify it. from multiprocessing import util, Pool, TimeoutError from multiprocessing.dummy import Pool as ThreadPool import threading import sys from functools import partial import time def worker(y): print("worker sleep {} sec, thread: {}".format(y, threading.current_thread())) start = time.time() while True: if time.time() - start >= y: break time.sleep(0.5) # show

问题 Below is my code , for which i am facing a multiprocessing issue. I see this question has been asked before and i have tried those solutions but it does not seem to work. Can someone help me out here ? from multiprocessing import Pool, Manager Class X: def _init_(): def method1(number1,var_a, var_b, var_c, var_d): return values if __name__ == 'main': for value in ["X", "Y"]: dict_values = Manager().dict() with Pool(1) as p: p.starmap(method1, [ (1, dict_values, var_a, var_b, var_c, var_d), (2

问题 I want to create a shared array with a dynamic size. I want to assign an array with an unknown size to it in another process. from multiprocessing import Process, Value, Array def f(a): b=[3,5,7] #resize(a,len(b)) # How to resize "a" ??? a[:]=b # Only works when "a" is initialized with the same size like the "b" arr = Array('d', 0) #Array with a size of 0 p = Process(target=f, args=(arr)) p.start() p.join() print arr[:] 回答1: The size of mp.Arrays can only be set once upon instantiation. You

问题 I want to create a shared array with a dynamic size. I want to assign an array with an unknown size to it in another process. from multiprocessing import Process, Value, Array def f(a): b=[3,5,7] #resize(a,len(b)) # How to resize "a" ??? a[:]=b # Only works when "a" is initialized with the same size like the "b" arr = Array('d', 0) #Array with a size of 0 p = Process(target=f, args=(arr)) p.start() p.join() print arr[:] 回答1: The size of mp.Arrays can only be set once upon instantiation. You