python-multithreading

TensorFlow/Keras multi-threaded model fitting

不问归期 提交于 2019-11-28 21:06:50
问题 I'm attempting to train multiple keras models with different parameter values using multiple threads (and the tensorflow backend). I've seen a few examples of using the same model within multiple threads, but in this particular case, I run into various errors regarding conflicting graphs, etc. Here's a simple example of what I'd like to be able to do: from concurrent.futures import ThreadPoolExecutor import numpy as np import tensorflow as tf from keras import backend as K from keras.layers

python asyncio, how to create and cancel tasks from another thread

天大地大妈咪最大 提交于 2019-11-28 17:16:09
I have a python multi-threaded application. I want to run an asyncio loop in a thread and post calbacks and coroutines to it from another thread. Should be easy but I cannot get my head around the asyncio stuff. I came up to the following solution which does half of what I want, feel free to comment on anything: import asyncio from threading import Thread class B(Thread): def __init__(self): Thread.__init__(self) self.loop = None def run(self): self.loop = asyncio.new_event_loop() asyncio.set_event_loop(self.loop) #why do I need that?? self.loop.run_forever() def stop(self): self.loop.call

Multithreading for Python Django

痞子三分冷 提交于 2019-11-28 16:21:40
Some functions should run asynchronously on the web server. Sending emails or data post-processing are typical use cases. What is the best (or most pythonic) way write a decorator function to run a function asynchronously? My setup is a common one: Python, Django, Gunicorn or Waitress, AWS EC2 standard Linux For example, here's a start: from threading import Thread def postpone(function): def decorator(*args, **kwargs): t = Thread(target = function, args=args, kwargs=kwargs) t.daemon = True t.start() return decorator desired usage: @postpone def foo(): pass #do stuff I've continued using this

Multiprocessing backed parallel loops cannot be nested below threads

大城市里の小女人 提交于 2019-11-28 11:06:15
What is the reason of such issue in joblib? 'Multiprocessing backed parallel loops cannot be nested below threads, setting n_jobs=1' What should I do to avoid such issue? Actually I need to implement XMLRPC server which run heavy computation in background thread and report current progress through polling from UI client. It uses scikit-learn which are based on joblib. P.S.: I've simply changed name of the thread to "MainThread" to avoid such warning and everything looks working good (run in parallel as expected without issues). What might be a problem in future for such workaround? This seems

Multi threading in Tkinter GUI, threads in different classes

倾然丶 夕夏残阳落幕 提交于 2019-11-28 08:37:57
问题 I'm currently learning the Tkinter GUI programming. And I'm stuck in somewhere in multi threading concept. Even though this topic is discussed several times here, I couldn't catch the concept and apply it to my small sample program. Below is my code: from PIL import Image, ImageTk from Tkinter import Tk, Label, BOTH from ttk import Frame, Style from Tkinter import * import time class Widgets(Frame): def __init__(self, parent): Frame.__init__(self, parent) self.grid() self.parent = parent self

Python IPC with matplotlib

ぐ巨炮叔叔 提交于 2019-11-28 06:40:38
问题 Project description: Connect existing "C" program (main control) to Python GUI/Widget. For this I'm using a FIFO. The C program is designed look at frame based telemetry. The Python GUI performs two functions: Runs/creates plots (probably created through matplotlib) via GUI widget as the user desires (individual .py files, scripts written by different users) Relays the frame number to the python plotting scripts after they have been created so they can "update" themselves after being given

Python Multiprocessing help exit on condition

允我心安 提交于 2019-11-28 05:42:29
问题 I'm breaking my teeth on multiprocessing within Python but I'm not having any luck wrapping my head around the subject. Basically I have a procedure that is time consuming to run. I need to run it for a range of 1 to 100 but I'd like to abort all processes once the condition I'm looking for has been met. The condition being the return value == 90. Here is a non multiprocess chunk of code. Can anyone give me an example of how they would convert it to a multiprocess function where the the code

Use of threading.Thread.join()

僤鯓⒐⒋嵵緔 提交于 2019-11-28 05:22:41
I am new to multithreading in python and trying to learn multithreading using threading module. I have made a very simple program of multi threading and i am having trouble understanding the threading.Thread.join method. Here is the source code of the program I have made import threading val = 0 def increment(): global val print "Inside increment" for x in range(100): val += 1 print "val is now {} ".format(val) thread1 = threading.Thread(target=increment, args=()) thread2 = threading.Thread(target=increment, args=()) thread1.start() #thread1.join() thread2.start() #thread2.join() What

Understanding thread.join(timeout)

ぃ、小莉子 提交于 2019-11-28 05:12:55
问题 So the timeout param, for a thread, should stop the thread after timeout seconds (if it hasn't terminated yet). In my software I'm trying to replace a Queue.Queue.join() (it contains an item for every thread: each thread will run Queue.Queue.task_done()) that could stop the software if a thread doesn't terminate. So if a thread, among other 50, doesn't terminate then it is all freezed. I want that every thread stops in 5 seconds , for example. So i will start each thread with timeout of 5

How terminate Python thread without checking flag continuously

怎甘沉沦 提交于 2019-11-28 04:49:42
问题 class My_Thread(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): print "Starting " + self.name cmd = [ "bash", 'process.sh'] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) for line in iter(p.stdout.readline, b''): print ("-- " + line.rstrip()) print "Exiting " + self.name def stop(self): print "Trying to stop thread " self.run = False thr = My_Thread() thr.start() time.sleep(30) thr.stop() thr.join() So i have thread like show