python pool apply_async and map_async do not block on full queue

Question

I am fairly new to python. I am using the multiprocessing module for reading lines of text on stdin, converting them in some way and writing them into a database. Here\'s a

Answer 1

apply_async returns an AsyncResult object, which you can wait on:

if len(batch) >= 10000:
    r = pool.apply_async(insert, args=(batch, i+1))
    r.wait()
    batch = []

Though if you want to do this in a cleaner manner, you should use a multiprocessing.Queue with a maxsize of 10000, and derive a Worker class from multiprocessing.Process that fetches from such a queue.