python threads - always have x active threads when iterating over n tasks

问题

What I basically want to do is the following:

import threading
import Queue

def test_thread(elem, q):
    q.put(elem ** 2)

a = [1,2,3,4,5,6,7,8]
q = Queue.Queue()
results = []
for x in range(8):
    print x
    threading.Thread(target=test_thread, args=(a[x], q)).start()
    results.append(q.get())

But instead of running all threads at once I want to run only say 2 in parallel adn iterate over the list. Once one thread is done the next value from the list should be processed. I could't find an example and I am not sure how to construct the loops for this.

In addition, I do not understand the behavior of the Queue. I would have expected that ALL squared numbers are in the queue. But instead there is only one value? (The code above has changed to store all results in "results"). Hints, comments, keywords are highly appreciated.

EDIT:

Second question:

Sorry, I thought q.get() will return all the results. But it just gives the elements in a queue like fashion.

回答1:

You can use a thread pool for this:

import threading
from multiprocessing.pool import ThreadPool

def test_thread(elem):
    return elem ** 2

a = [1,2,3,4,5,6,7,8]
pool = ThreadPool(2) # 2 worker threads
results = []
for x in range(8):
    print x
    results.append(pool.apply_async(test_thread, args=(a[x],)))

results = [result.get() for result in results]
# You can also replace this for loop altogether using pool.map
# and get the same result:
# results = pool.map(test_thread, range(8))
print(results)

Output:

0
1
2
3
4
5
6
7
[1, 4, 9, 16, 25, 36, 49, 64]

The ThreadPool class is a mostly undocumented part of the multiprocessing module. It can also be access via multiprocessing.dummy.Pool. It allows you to create a pool of threads to handle any number of work items, while always limiting the number of work items being concurrently processed to something you specify. You can use the documentation for the normal multiprocessing.Pool to learn about its API. It's exactly the same, except everywhere it says "process", you replace it with "thread".

I'm not sure I follow the second part of your question about Queue.Queue. On each iteration of your for loop, you're putting one item into the Queue inside test_thread, and then consuming it inside the for loop using results.append(q.get()). So while there is never more than one item in the Queue at a time, it is being used to transfer all the values that end up in the results list - one for each item in the range(8) list.

来源：https://stackoverflow.com/questions/25875398/python-threads-always-have-x-active-threads-when-iterating-over-n-tasks

标签

python

multithreading

list

queue