Write data to disk in Python as a background process

Question

I have a program in Python that basically does the following:

for j in xrange(200):
    # 1) Compute a bunch of data
    # 2) Write data to disk

<

Answer 1

Simple way would be to use just threading and the queue. On the other hand, if the computing part does not depend on global state, and you have machine with multiple CPU cores, more efficient way would be to use process pool

from multiprocessing import Pool

def compute_data(x):
    return some_calculation_with(x)

if __name__ == '__main__':
    pool = Pool(processes=4) # let's say you have quad-core, so start 4 workers

    with open("output_file","w") as outfile:
        for calculation_result in pool.imap(compute_data, range(200)):
        # pool.imap returns results as they come from process pool    
            outfile.write(calculation_result)