How to average summaries over multiple batches?

Question

Assuming I have a bunch of summaries defined like:

loss = ...
tf.scalar_summary(\"loss\", loss)
# ...
summaries = tf.m         


        

Answer 1

I would avoid calculating the average outside the graph.

You can use tf.train.ExponentialMovingAverage:

ema = tf.train.ExponentialMovingAverage(decay=my_decay_value, zero_debias=True)
maintain_ema_op = ema.apply(your_losses_list)

# Create an op that will update the moving averages after each training step.
with tf.control_dependencies([your_original_train_op]):
    train_op = tf.group(maintain_ema_op)

Then, use:

sess.run(train_op)

That will call maintain_ema_op because it is defined as a control dependency.

In order to get your exponential moving averages, use:

moving_average = ema.average(an_item_from_your_losses_list_above)

And retrieve its value using:

value = sess.run(moving_average)

This calculates the moving average within your calculation graph.