python pool apply_async and map_async do not block on full queue

Question

I am fairly new to python. I am using the multiprocessing module for reading lines of text on stdin, converting them in some way and writing them into a database. Here\'s a

Answer 1

Not pretty, but you can access the internal queue size and wait until it's below your maximum desired size before adding new items:

max_pool_queue_size = 20

for i in range(10000):
  pool.apply_async(some_func, args=(...))

  while pool._taskqueue.qsize() > max_pool_queue_size:
    time.sleep(1)