python-multithreading

class My_Thread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): print "Starting " + self.name cmd = [ "bash", 'process.sh'] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in iter(p.stdout.readline, b''): print ("-- " + line.rstrip()) print "Exiting " + self.name def stop(self): print "Trying to stop thread " self.run = False thr = My_Thread() thr.start() time.sleep(30) thr.stop() thr.join() So i have thread like show above, actually work on windows and process.sh is bash script which run in cygwin and takes around 5

I'm building a web service for iterative batch processing of data using CherryPy. The ideal workflow is as follows: Users POST data to the service for processing When the processing job is free, it collects the queued data and starts another iteration While the job is processing, users are POSTing more data to the queue for the next iteration Once the current iteration is finished, the results are passed back so that users can GET them using the same API. The job starts again with the next batch of queued data. The key consideration here is that the processing should run as fast as possible

I have three commands that would otherwise be easily chained together on the command-line like so: $ echo foo | firstCommand - | secondCommand - | thirdCommand - > finalOutput In other words, the firstCommand processes foo from standard input and pipes the result to secondCommand , which in turn processes that input and pipes its output to thirdCommand , which does processing and redirects its output to the file finalOutput . I have been trying to recapitulate this in a Python script, using threading. I'd like to use Python in order to manipulate the output from firstCommand before passing it

I would like to create either a Thread or a Process which runs forever in a While True loop. I need to send and receive data to the worker in the form for queues, either a multiprocessing.Queue() or a collections.deque(). I prefer to use collections.deque() as it is significantly faster. I also need to be able to kill the worker eventually (as it runs in a while True loop. Here is some test code I've put together to try and understand the differences between Threads, Processes, Queues, and deque .. import time from multiprocessing import Process, Queue from threading import Thread from

I want to pass named arguments to the target function, while creating a Thread object. Following is the code that I have written: import threading def f(x=None, y=None): print x,y t = threading.Thread(target=f, args=(x=1,y=2,)) t.start() I get a syntax error for "x=1", in Line 6. I want to know how I can pass keyword arguments to the target function. t = threading.Thread(target=f, kwargs={'x': 1,'y': 2}) this will pass a dictionary with the keyword arguments' names as keys and argument values as values in the dictionary. the other answer above won't work, because the "x" and "y" are undefined

I am about to write some computationally-intensive Python code that'll almost certainly spend most of its time inside numpy 's linear algebra functions. The problem at hand is embarrassingly parallel . Long story short, the easiest way for me to take advantage of that would be by using multiple threads. The main barrier is almost certainly going to be the Global Interpreter Lock (GIL). To help design this, it would be useful to have a mental model for which numpy operations can be expected to release the GIL for their duration. To this end, I'd appreciate any rules of thumb, dos and don'ts,

import threading threads = [] for n in range(0, 60000): t = threading.Thread(target=function,args=(x, n)) t.start() threads.append(t) for t in threads: t.join() It is working well for range up to 800 on my laptop, but if I increase range to more than 800 I get the error can't create new thread . How can I control number to threads to get created or any other way to make it work like timeout? I tried using threading.BoundedSemaphore function but that doesn't seem to work properly. The problem is that no major platform (as of mid-2013) will let you create anywhere near this number of threads.