python-multiprocessing

问题 I am looking for a dead simple example on how to share a complex object between two or more processes in python. On my main i have something like if __name__ == "__main__": FirstClass().start() SecondClass().start() etc... and each class is defined like: class FirstClass(multiprocessing.Process): def __init__(self): super(FirstClass, self).__init__() [...] I would like to have a MySharedClass with inside all the data i need (list of custom objects and so on) that i can access and modify from

问题 Context A Keras model (link here, for the sake of MWE) needs to predict a lot of test data, in parallel. I define a cube as a 3D numpy.ndarray of uint . Each of its vertical slices is a column , which is npixels = 128 height, nbins = 128 depth. Each prediction transforms a column in a denoised column (same size). I'm providing three approaches: a single-threaded, a multiprocessing and a pathos package multiprocessing. Both the multi-threaded approaches are not working, and I don't get the

问题 I'm using slurm to manage some of our calculations but sometimes jobs are getting killed with an out-of-memory error even though this should not be the case. This strange issue has been with python jobs using multiprocessing in particular. Here's a minimal example to reproduce this behavior #!/usr/bin/python from time import sleep nmem = int(3e7) # this will amount to ~1GB of numbers nprocs = 200 # will create this many workers later nsleep = 5 # sleep seconds array = list(range(nmem)) #

问题 Hi guys i'm fairly new in python. what i'm trying to do is to move my old code into multiprocessing however i'm facing some errors that i hope anyone could help me out. My code is used to check a few thousand links given in a text form to check for certain tags. Once found it will output it to me. Due to the reason i have a few thousand links to check, speed is an issue and hence the need for me to move to multi processing. Update: i'm having return errors of HTTP 503 errors. Am i sending too

问题 Ignoring the actual utility, I have the following classes: from multiprocessing import Manager import threading class MgrStaticTarget(object): def __init__(self): self._mgr = Manager() self._thd = threading.Thread(target=self.async_targ) @staticmethod def async_targ(): pass class MgrInstanceTarget(object): def __init__(self): self._mgr = Manager() self._thd = threading.Thread(target=self.async_targ) def async_targ(self): pass and I'm observing behavior I don't understand. Upon repeated

问题 I am new to python and i am trying to scale my processing in parallel. I have a file with certain number of tuples, each with certain value in the last column. I want to split this file data and apply my function in parallel to each chunk. But the thing is to split the data in to chunks based on last column value and apply the function for each chunk. For example, last column may have 'a' for some tuples and 'b' for some and 'c' for some. So in that case, i should get three chunks and process

问题 I spawn few processes using the Python multiprocessing module. However when I call netstat -nptl, each ip:port listeners listed under the same PID. I'm using Python 2.7 on Ubuntu 14.04. netstat -V >> net-tools 1.60 >> netstat 1.42 (2001-04-15) Relevant code: import unittest import multiprocessing import socket import os import time import ex1 class Listener(multiprocessing.Process): def __init__(self, _ttl): super(Listener, self).__init__() self.ttl = _ttl self.socket = socket.socket(socket

问题 I'm trying to implement a function that uses python multiprocessing in order to speed-up a calculation. I'm trying to create a pairwise distance matrix but the implementation with for loops takes more than 8 hours. This code seems to work faster but when I print the matrix is full of zeros. When I print the rows in the function it seems to work. I think is a scope problem but I cannot understand how to deal with it. import multiprocessing import time import numpy as np def MultiProcessedFunc

问题 This seems to be a python 3.6 bug When assigning a manager.Queue to a manager.Namespace attribute one gets an TypeError . To reproduce it is simple: >>> from multiprocessing import Manager >>> manager = Manager() >>> np = manager.Namespace() >>> np.shared_queue = manager.Queue() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\users\leonardo.schettini\appdata\local\programs\python\python36-32\Lib\multiprocessing\managers.py", line 1065, in __setattr__ return

问题 In one answer to: Is shared readonly data copied to different processes for multiprocessing? a working solution for shared memory for a numpy array is given. How would the same look like if a pandas DataFrame should be used? Background: I would like to be able to write to the DataFrame during multiprocessing and would like to be able to process it further after the multiprocessing has finished. 来源： https://stackoverflow.com/questions/53320422/how-to-use-pandas-dataframe-in-shared-memory