python-multithreading

Pandas, Concurrent.Futures and the GIL

对着背影说爱祢 提交于 2020-05-13 14:12:01
问题 I'm writing code using Pandas 0.18/Python 3.5 on an intel i3 (four cores). I have read this: https://www.continuum.io/content/pandas-releasing-gil I also have some work that is IO bound (parsing CSV files into dataframes). I have to do a lot of calculation that is mostly multiplying dataframes. My code is currently parallel using concurrent.futures ThreadPoolExecutor . My question is: In general, should I be using threads to run pandas jobs in parallel, or does pandas make effective use of

run multiple instances of python script simultaneously

拈花ヽ惹草 提交于 2020-05-09 18:32:30
问题 I am trying to create 86 instances of task.py to run simultaneously. import sys import subprocess for file in range(86): subprocess.call([sys.executable,'task.py',str(file)+'in.csv',str(filen)+'out.csv']) 回答1: subprocess.call waits for command to complete. Use subprocess.Popen instead: import sys import subprocess procs = [] for i in range(86): proc = subprocess.Popen([sys.executable, 'task.py', '{}in.csv'.format(i), '{}out.csv'.format(i)]) procs.append(proc) for proc in procs: proc.wait() 来源

run multiple instances of python script simultaneously

醉酒当歌 提交于 2020-05-09 18:32:14
问题 I am trying to create 86 instances of task.py to run simultaneously. import sys import subprocess for file in range(86): subprocess.call([sys.executable,'task.py',str(file)+'in.csv',str(filen)+'out.csv']) 回答1: subprocess.call waits for command to complete. Use subprocess.Popen instead: import sys import subprocess procs = [] for i in range(86): proc = subprocess.Popen([sys.executable, 'task.py', '{}in.csv'.format(i), '{}out.csv'.format(i)]) procs.append(proc) for proc in procs: proc.wait() 来源

Matplotlib set_data makes my program freeze (please check the 'Updated issue')

柔情痞子 提交于 2020-05-09 11:54:27
问题 I am trying to plot two updating plots, one is a diagram and the other one is an image captured from the camera. I get an error on this line: "current_line.set_data(y_data)" in the "update" function. The error says: "AttributeError: 'list' object has no attribute 'set_data'". Any idea why am I getting this error? If I comment out this line I will get changing images from the camera and everything except the second plot seems fine (because the second plot is not updating) but I need the second

Matplotlib set_data makes my program freeze (please check the 'Updated issue')

我们两清 提交于 2020-05-09 11:54:12
问题 I am trying to plot two updating plots, one is a diagram and the other one is an image captured from the camera. I get an error on this line: "current_line.set_data(y_data)" in the "update" function. The error says: "AttributeError: 'list' object has no attribute 'set_data'". Any idea why am I getting this error? If I comment out this line I will get changing images from the camera and everything except the second plot seems fine (because the second plot is not updating) but I need the second

python multiprocessing pool timeout

别说谁变了你拦得住时间么 提交于 2020-04-05 06:28:02
问题 I want to use multiprocessing.Pool, but multiprocessing.Pool can't abort a task after a timeout. I found solution and some modify it. from multiprocessing import util, Pool, TimeoutError from multiprocessing.dummy import Pool as ThreadPool import threading import sys from functools import partial import time def worker(y): print("worker sleep {} sec, thread: {}".format(y, threading.current_thread())) start = time.time() while True: if time.time() - start >= y: break time.sleep(0.5) # show

how to start and stop recording with single button

无人久伴 提交于 2020-03-05 06:06:11
问题 I am having an issue with this code, when I try to run this code with two separate buttons it runs fine but when I try to use only a single button to start and stop the recording by changing its text(label) to start and stop recording the code does not work. The text of the button changes but it does not record anything. import tkinter as tk import threading import pyaudio import wave from tkinter import * import tkinter.font as font from tkinter.filedialog import asksaveasfile class App():

How to share state when using concurrent futures

点点圈 提交于 2020-03-01 18:33:27
问题 I am aware using the traditional multiprocessing library I can declare a value and share the state between processes. https://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#sharing-state-between-processes When using the newer concurrent.futures library how can I share state between my processes? import concurrent.futures def get_user_object(batch): # do some work counter = counter + 1 print(counter) def do_multithreading(batches): with concurrent.futures

How to share state when using concurrent futures

孤街浪徒 提交于 2020-03-01 18:33:23
问题 I am aware using the traditional multiprocessing library I can declare a value and share the state between processes. https://docs.python.org/3/library/multiprocessing.html?highlight=multiprocessing#sharing-state-between-processes When using the newer concurrent.futures library how can I share state between my processes? import concurrent.futures def get_user_object(batch): # do some work counter = counter + 1 print(counter) def do_multithreading(batches): with concurrent.futures

Variable passed to multiple threads of the same function

旧巷老猫 提交于 2020-02-25 06:41:05
问题 I am currently working on a system that sends webhooks to multiple Discord using dhooks. This is the function used to call the function which sends the webhooks. The length of webhooks is 2 and it's a list of lists. for hook in webhooks: thread.start_new_thread(send_hook,(hook, embed, tag)) This is the send_hook function: def send_hook(hook, embed, tag): embed.set_footer(text="{} x Will".format(hook[0])) <-- This is the part where the error happens webhook = Webhook(hook[1]) webhook.send