python-multiprocessing

I am writing a simple code of cropping images and saving it. But the problem is that the number of images is about 150,000+ and I want to improve the speed. So, at first I wrote a code with simple for loops, like the following: import cv2 import numpy import sys textfile=sys.argv[1] file_list=open(textfile) files=file_list.read().split('\n') idx=0 for eachfile in files: image=cv2.imread(eachfile) idx+=1 if image is None: pass outName=eachfile.replace('/data','/changed_data') if image.shape[0]==256: image1=image[120:170,120:170] elif image.shape[0]==50: image1=image cv2.imwrite(outName,image1)

I'm actually finding it hard to believe that I've run into the issue I have, it seems like it would be a big bug in the python multiprocessing module... Anyways the problem I've run into is that whenever I pass a multiprocessing.Queue to a multiprocessing.Pool worker as an argument the pool worker never executes its code. I've been able to reproduce this bug even on a very simple test that is a slightly modified version of example code found in the python docs . Here is the original version of the example code for Queues: from multiprocessing import Process, Queue def f(q): q.put([42, None,

I am trying to use this example as a template for a queuing system on my cherrypy app. I was able to convert it from python 2 to python 3 (change from Queue import Empty into from queue import Empty ) and to execute it in Ubuntu. But when I execute it in Windows I get the following error: F:\workspace\test>python test.py Traceback (most recent call last): File "test.py", line 112, in <module> broker.start() File "C:\Anaconda3\lib\multiprocessing\process.py", line 105, in start self._popen = self._Popen(self) File "C:\Anaconda3\lib\multiprocessing\context.py", line 212, in _Popen return

Why does the following Python code using the concurrent.futures module hang forever? import concurrent.futures class A: def f(self): print("called") class B(A): def f(self): executor = concurrent.futures.ProcessPoolExecutor(max_workers=2) executor.submit(super().f) if __name__ == "__main__": B().f() The call raises an invisible exception [Errno 24] Too many open files (to see it, replace the line executor.submit(super().f) with print(executor.submit(super().f).exception()) ). However, replacing ProcessPoolExecutor with ThreadPoolExecutor prints "called" as expected. Why does the following

First off I wasn't sure if I should post this as a Ubuntu question or here. But I'm guessing it's more of an Python question than a OS one. My Python application is running on top of Ubuntu on a 64 core AMD server. It pulls images from 5 GigE cameras over the network by calling out to a .so through ctypes and then processes them. I am seeing frequent pauses in my application causing frames from the cameras to be dropped by the external camera library. To debug this I've used the popular psutil Python package with which I log out CPU stats every 0.2 seconds in a separate thread. I sleep for 0.2

I'm trying to use Keras to run a reinforcement learning algorithm. In this algorithm, I'm training a neural network. What's different from other learning problems is that I need to use the neural network itself to generate training data, and repeat this after it updates. I run into problem when I'm trying to generate training data in parallel. The problem is that I can't tell Theano to use GPU while training because it will also use GPU when generating training data and cause problems if invoked by multiple processes. What's more, I Theano wont run in multi-thread mode even when I write THEANO