python-multithreading

Matplotlib: simultaneous plotting in multiple threads

我们两清 提交于 2019-11-28 03:29:50
I am trying to do some plotting in parallel to finish large batch jobs quicker. To this end, I start a thread for each plot I plan on making. I had hoped that each thread would finish its plotting and close itself (as I understand it, Python closes threads when they get through all the statements in run()). Below is some code that shows this behavior. If the line that creates a figure is commented out, it runs as expected. Another plausibly helpful tidbit is that it also runs as expected when you only spawn one thread. import matplotlib.pyplot as plt import time import Queue import threading

How to best perform Multiprocessing within requests with the python Tornado server?

女生的网名这么多〃 提交于 2019-11-28 02:51:47
I am using the I/O non-blocking python server Tornado. I have a class of GET requests which may take a significant amount of time to complete (think in the range of 5-10 seconds). The problem is that Tornado blocks on these requests so that subsequent fast requests are held up until the slow request completes. I looked at: https://github.com/facebook/tornado/wiki/Threading-and-concurrency and came to the conclusion that I wanted some combination of #3 (other processes) and #4 (other threads). #4 on its own had issues and I was unable to get reliable control back to the ioloop when there was

How to return data from a CherryPy BackgroundTask running as fast as possible

倖福魔咒の 提交于 2019-11-28 02:02:43
问题 I'm building a web service for iterative batch processing of data using CherryPy. The ideal workflow is as follows: Users POST data to the service for processing When the processing job is free, it collects the queued data and starts another iteration While the job is processing, users are POSTing more data to the queue for the next iteration Once the current iteration is finished, the results are passed back so that users can GET them using the same API. The job starts again with the next

How to thread multiple subprocess instances in Python 2.7?

跟風遠走 提交于 2019-11-28 01:51:58
问题 I have three commands that would otherwise be easily chained together on the command-line like so: $ echo foo | firstCommand - | secondCommand - | thirdCommand - > finalOutput In other words, the firstCommand processes foo from standard input and pipes the result to secondCommand , which in turn processes that input and pipes its output to thirdCommand , which does processing and redirects its output to the file finalOutput . I have been trying to recapitulate this in a Python script, using

Timer cannot restart after it is being stopped in Python

╄→尐↘猪︶ㄣ 提交于 2019-11-28 01:38:38
问题 I am using Python 2.7. I have a timer that keeps repeating a timer callback action until it has been stopped. It uses a Timer object. The problem is that after it has been stopped, it cannot be restarted. The Timer object code is as follows; from threading import Timer class RepeatingTimer(object): def __init__(self,interval, function, *args, **kwargs): super(RepeatingTimer, self).__init__() self.args = args self.kwargs = kwargs self.function = function self.interval = interval def start(self

Cancellable threading.Timer in Python

我们两清 提交于 2019-11-27 18:31:36
I am trying to write a method that counts down to a given time and unless a restart command is given, it will execute the task. But I don't think Python threading.Timer class allows for timer to be cancelable. import threading def countdown(action): def printText(): print 'hello!' t = threading.Timer(5.0, printText) if (action == 'reset'): t.cancel() t.start() I know the above code is wrong somehow. Would appreciate some kind guidance over here. Honest Abe You would call the cancel method after you start the timer: import time import threading def hello(): print "hello, world" time.sleep(2) t

How to Multi-thread an Operation Within a Loop in Python

北城以北 提交于 2019-11-27 17:23:31
Say I have a very large list and I'm performing an operation like so: for item in items: try: api.my_operation(item) except: print 'error with item' My issue is two fold: There are a lot of items api.my_operation takes forever to return I'd like to use multi-threading to spin up a bunch of api.my_operations at once so I can process maybe 5 or 10 or even 100 items at once. If my_operation() returns an exception (because maybe I already processed that item) - that's OK. It won't break anything. The loop can continue to the next item. Note : this is for Python 2.7.3 First, in Python, if your code

Pass keyword arguments to target function in Python threading.Thread

北战南征 提交于 2019-11-27 17:07:58
问题 I want to pass named arguments to the target function, while creating a Thread object. Following is the code that I have written: import threading def f(x=None, y=None): print x,y t = threading.Thread(target=f, args=(x=1,y=2,)) t.start() I get a syntax error for "x=1", in Line 6. I want to know how I can pass keyword arguments to the target function. 回答1: t = threading.Thread(target=f, kwargs={'x': 1,'y': 2}) this will pass a dictionary with the keyword arguments' names as keys and argument

numpy and Global Interpreter Lock

非 Y 不嫁゛ 提交于 2019-11-27 15:44:50
问题 I am about to write some computationally-intensive Python code that'll almost certainly spend most of its time inside numpy 's linear algebra functions. The problem at hand is embarrassingly parallel. Long story short, the easiest way for me to take advantage of that would be by using multiple threads. The main barrier is almost certainly going to be the Global Interpreter Lock (GIL). To help design this, it would be useful to have a mental model for which numpy operations can be expected to

Can't create new threads in Python

99封情书 提交于 2019-11-27 15:31:09
问题 import threading threads = [] for n in range(0, 60000): t = threading.Thread(target=function,args=(x, n)) t.start() threads.append(t) for t in threads: t.join() It is working well for range up to 800 on my laptop, but if I increase range to more than 800 I get the error can't create new thread . How can I control number to threads to get created or any other way to make it work like timeout? I tried using threading.BoundedSemaphore function but that doesn't seem to work properly. 回答1: The