concurrent.futures

I am trying to send information extracted from lines of a big file to a process running on some server. To speed this up, I would like to do this with some threads in parallel. Using the Python 2.7 backport of concurrent.futures I tried this: f = open("big_file") with ThreadPoolExecutor(max_workers=4) as e: for line in f: e.submit(send_line_function, line) f.close() However, this is problematic, because all futures get submitted instantly, so that my machine runs out of memory, because the complete file gets loaded into memory. My question is, if there is an easy way to only submit a new

In the ThreadPoolExecutor (TPE), is the callback always guaranteed to run in the same thread as the submitted function? For example, I tested this with the following code. I ran it many times and it seemed like func and callback always ran in the same thread. import concurrent.futures import random import threading import time executor = concurrent.futures.ThreadPoolExecutor(max_workers=3) def func(x): time.sleep(random.random()) return threading.current_thread().name def callback(future): time.sleep(random.random()) x = future.result() cur_thread = threading.current_thread().name if (cur

A simple way of implementing multiprocessing in python is from multiprocessing import Pool def calculate(number): return number if __name__ == '__main__': pool = Pool() result = pool.map(calculate, range(4)) An alternative implementation based on futures is from concurrent.futures import ProcessPoolExecutor def calculate(number): return number with ProcessPoolExecutor() as executor: result = executor.map(calculate, range(4)) Both alternatives do essentially the same thing, but one striking difference is that we don't have to guard the code with the usual if __name__ == '__main__' clause. Is

I was just searching about this new implementation, and i use python 2.7, i must install this , so if i use it, i'll forget the word GIL on CPython? No, concurrent.futures has almost nothing whatsoever to do with the GIL. Using processes instead of threads is medicine for the GIL. (Of course, like all medicine, it has side effects. But it works.) The futures module just gives you a simpler way to schedule and wait on tasks than using threading or multiprocessing directly. And it has the added advantage that you can swap between a thread pool and a process pool (and maybe even a greenlet loop,

To clarify the reason for this question: It is confusing to use two modules with the same name. What do they represent that makes them distinct? What task(s) can one solve that the other can't and vice-versa? The asyncio documentation covers the differences: class asyncio.Future(*, loop=None) This class is almost compatible with concurrent.futures.Future . Differences: result() and exception() do not take a timeout argument and raise an exception when the future isn’t done yet. Callbacks registered with add_done_callback() are always called via the event loop’s call_soon_threadsafe() . This