How to check NaN in gradients in Tensorflow when updating?

Question

All,

When you train a large model with large amount samples, some samples may be cause NaN gradient when parameter updating.

And I want to find these samples

Answer 1

You could use tf.is_nan in combination with tf.cond to only execute the rest of your code if the loss is not NaN.