How to fix the python multiprocessing matplotlib savefig() issue?

Question

I want to speed up matplotlib.savefig() for many figures by multiprocessing module, and trying to benchmark the performance between parallel and sequence.

Below is t

Answer 1

You can try to move all of the matplotlib code(including the import) to a function.

1. Make sure you don't have a import matplotlib or import matplotlib.pyplot as plt at the top of your code.

2. create a function that does all the matplotlib including the import.

Example:

import numpy as np
from multiprocessing import pool

def graphing_function(graph_data):
    import matplotlib.pyplot as plt
    plt.figure()
    plt.hist(graph_data.data)
    plt.savefig(graph_data.filename)
    plt.close()
    return

pool = Pool(4)
pool.map(graphing_function, data_list) 

Answer 2

It is not really a bug, per-say, more of a limitation.

The explanation is in the last line of your error mesage:

PicklingError: Can't pickle <function notify_axes_change at 0x029F5030>: it's not found as matplotlib.backends.backend_qt4.notify_axes_change

It is telling you that elements of the figure objects can not be pickled, which is how MultiProcess passes data between the processes. The objects are pickled in the main processes, shipped as pickles, and then re-constructed on the other side. Even if you fixed this exact issue (maybe by using a different backend, or stripping off the offending function (which might break things in other ways)) I am pretty sure there are core parts of Figure, Axes, or Canvas objects that can not be pickled.

As @bigbug point to, an example of how to get around this limitation, Matplotlib: simultaneous plotting in multiple threads. The basic idea is that you push your entire plotting routine off to the sub-process so you only push numpy arrays an maybe some configuration information across the process boundry.