Matplotlib: simultaneous plotting in multiple threads

Question

I am trying to do some plotting in parallel to finish large batch jobs quicker. To this end, I start a thread for each plot I plan on making.

I had hoped that each thread would finish its plotting and close itself (as I understand it, Python closes threads when they get through all the statements in run()). Below is some code that shows this behavior.

If the line that creates a figure is commented out, it runs as expected. Another plausibly helpful tidbit is that it also runs as expected when you only spawn one thread.

import matplotlib.pyplot as plt
import time
import Queue
import threading

def TapHistplots():
    ##  for item in ['str1']:
# # it behaves as expected if the line above is used instead of the one below
    for item in ['str1','str2']:
        otheritem = 1
        TapHistQueue.put((item, otheritem))
        makeTapHist().start()

class makeTapHist(threading.Thread):
    def run(self):
        item, otheritem = TapHistQueue.get()
        fig = FigureQueue.get()
        FigureQueue.put(fig+1)
        print item+':'+str(fig)+'\n',
        time.sleep(1.3)
        plt.figure(fig) # comment out this line and it behaves as expected
        plt.close(fig)

TapHistQueue = Queue.Queue(0)
FigureQueue = Queue.Queue(0)
def main():
    start = time.time()
    """Code in here runs only when this module is run directly"""
    FigureQueue.put(1)
    TapHistplots()
    while threading.activeCount()>1:
        time.sleep(1)
        print 'waiting on %d threads\n' % (threading.activeCount()-1),
    print '%ds elapsed' % (time.time()-start)

if __name__ == '__main__':
    main()

Any help is duly appreciated.

Answer 1

Why not just use multiprocessing? As far as I can tell from your description, threading won't help you much, anyway...

Matplotlib already threads so that you can display and interact with multiple figures at once. If you want to speed up batch processing on a multicore machine, you're going to need multiprocessing regardless.

As a basic example (Warning: This will create 20 small .png files in whatever directory you run it in!)

import multiprocessing
import matplotlib.pyplot as plt
import numpy as np

def main():
    pool = multiprocessing.Pool()
    num_figs = 20
    input = zip(np.random.randint(10,1000,num_figs), 
                range(num_figs))
    pool.map(plot, input)

def plot(args):
    num, i = args
    fig = plt.figure()
    data = np.random.randn(num).cumsum()
    plt.plot(data)
    plt.title('Plot of a %i-element brownian noise sequence' % num)
    fig.savefig('temp_fig_%02i.png' % i)

main()

Answer 2

For pylab interface there is a solution Asynchronous plotting with threads.

Without pylab there could be different solutions for each matplotlib's backends (Qt, GTK, WX, Tk). The problem is that each GUI toolkit has each own GUI mainloop. You could see how ipython deals with it.