Tensorflow: How to Display Custom Images in Tensorboard (e.g. Matplotlib Plots)

Question

The Image Dashboard section of the Tensorboard ReadMe says:

Since the image dashboard supports arbitrary pngs, you can use this to embed custom visualizations (e.g. matplotlib scatterplots) into TensorBoard.

I see how a pyplot image could be written to file, read back in as a tensor, and then used with tf.image_summary() to write it to TensorBoard, but this statement from the readme suggests there is a more direct way. Is there? If so, is there any further documentation and/or examples of how to do this efficiently?

Answer 1

It is quite easy to do if you have the image in a memory buffer. Below, I show an example, where a pyplot is saved to a buffer and then converted to a TF image representation which is then sent to an image summary.

import io import matplotlib.pyplot as plt import tensorflow as tf   def gen_plot():     """Create a pyplot plot and save to buffer."""     plt.figure()     plt.plot([1, 2])     plt.title("test")     buf = io.BytesIO()     plt.savefig(buf, format='png')     buf.seek(0)     return buf   # Prepare the plot plot_buf = gen_plot()  # Convert PNG buffer to TF image image = tf.image.decode_png(plot_buf.getvalue(), channels=4)  # Add the batch dimension image = tf.expand_dims(image, 0)  # Add image summary summary_op = tf.summary.image("plot", image)  # Session with tf.Session() as sess:     # Run     summary = sess.run(summary_op)     # Write summary     writer = tf.train.SummaryWriter('./logs')     writer.add_summary(summary)     writer.close() 

This gives the following TensorBoard visualization:

Answer 2

A bit late with my answer. With tf-matplotlib a simple scatter plot boils down to:

import tensorflow as tf import numpy as np  import tfmpl  @tfmpl.figure_tensor def draw_scatter(scaled, colors):      '''Draw scatter plots. One for each color.'''       figs = tfmpl.create_figures(len(colors), figsize=(4,4))     for idx, f in enumerate(figs):         ax = f.add_subplot(111)         ax.axis('off')         ax.scatter(scaled[:, 0], scaled[:, 1], c=colors[idx])         f.tight_layout()      return figs  with tf.Session(graph=tf.Graph()) as sess:      # A point cloud that can be scaled by the user     points = tf.constant(         np.random.normal(loc=0.0, scale=1.0, size=(100, 2)).astype(np.float32)     )     scale = tf.placeholder(tf.float32)             scaled = points*scale      # Note, `scaled` above is a tensor. Its being passed `draw_scatter` below.      # However, when `draw_scatter` is invoked, the tensor will be evaluated and a     # numpy array representing its content is provided.        image_tensor = draw_scatter(scaled, ['r', 'g'])     image_summary = tf.summary.image('scatter', image_tensor)           all_summaries = tf.summary.merge_all()       writer = tf.summary.FileWriter('log', sess.graph)     summary = sess.run(all_summaries, feed_dict={scale: 2.})     writer.add_summary(summary, global_step=0) 

When executed, this results in the following plot inside Tensorboard

Note that tf-matplotlib takes care about evaluating any tensor inputs, avoids pyplot threading issues and supports blitting for runtime critical plotting.

Answer 3

Next script does not use intermediate RGB/PNG encoding. It also fixes the issue with additional operation construction during execution, single summary is reused.

Size of the figure is expected to remain the same during execution

Solution that works:

import matplotlib.pyplot as plt import tensorflow as tf import numpy as np  def get_figure():   fig = plt.figure(num=0, figsize=(6, 4), dpi=300)   fig.clf()   return fig   def fig2rgb_array(fig, expand=True):   fig.canvas.draw()   buf = fig.canvas.tostring_rgb()   ncols, nrows = fig.canvas.get_width_height()   shape = (nrows, ncols, 3) if not expand else (1, nrows, ncols, 3)   return np.fromstring(buf, dtype=np.uint8).reshape(shape)   def figure_to_summary(fig):   image = fig2rgb_array(fig)   summary_writer.add_summary(     vis_summary.eval(feed_dict={vis_placeholder: image}))   if __name__ == '__main__':       # construct graph       x = tf.Variable(initial_value=tf.random_uniform((2, 10)))       inc = x.assign(x + 1)        # construct summary       fig = get_figure()       vis_placeholder = tf.placeholder(tf.uint8, fig2rgb_array(fig).shape)       vis_summary = tf.summary.image('custom', vis_placeholder)        with tf.Session() as sess:         tf.global_variables_initializer().run()         summary_writer = tf.summary.FileWriter('./tmp', sess.graph)          for i in range(100):           # execute step           _, values = sess.run([inc, x])           # draw on the plot           fig = get_figure()           plt.subplot('111').scatter(values[0], values[1])           # save the summary           figure_to_summary(fig) 

Answer 4

This intends to complete Andrzej Pronobis' answer. Following closely his nice post, I set up this minimal working example:

    plt.figure()     plt.plot([1, 2])     plt.title("test")     buf = io.BytesIO()     plt.savefig(buf, format='png')     buf.seek(0)     image = tf.image.decode_png(buf.getvalue(), channels=4)     image = tf.expand_dims(image, 0)     summary = tf.summary.image("test", image, max_outputs=1)     writer.add_summary(summary, step) 

Where writer is an instance of tf.summary.FileWriter. This gave me the following error: AttributeError: 'Tensor' object has no attribute 'value' For which this github post had the solution: the summary has to be evaluated (converted into a string) before being added to the writer. So the working code for me remained as follows (simply add the .eval() call in the last line):

    plt.figure()     plt.plot([1, 2])     plt.title("test")     buf = io.BytesIO()     plt.savefig(buf, format='png')     buf.seek(0)     image = tf.image.decode_png(buf.getvalue(), channels=4)     image = tf.expand_dims(image, 0)     summary = tf.summary.image("test", image, max_outputs=1)     writer.add_summary(summary.eval(), step) 

This could be short enough to be a comment on his answer, but these can be easily overlooked (and I may be doing something else different too), so here it is, hope it helps!

Cheers,
Andres