Deep-Learning Nan loss reasons

Question

Perhaps too general a question, but can anyone explain what would cause a Convolutional Neural Network to diverge?

Specifics:

I am using Tensorflow\'s iris_tra

Answer 1

The reason for nan, inf or -inf often comes from the fact that division by 0.0 in TensorFlow doesn't result in a division by zero exception. It could result in a nan, inf or -inf "value". In your training data you might have 0.0 and thus in your loss function it could happen that you perform a division by 0.0.

a = tf.constant([2., 0., -2.])
b = tf.constant([0., 0., 0.])
c = tf.constant([1., 1., 1.])
print((a / b) + c)

Output is the following tensor:

tf.Tensor([ inf  nan -inf], shape=(3,), dtype=float32)

Adding a small eplison (e.g., 1e-5) often does the trick. Additionally, since TensorFlow 2 the opteration tf.math.division_no_nan is defined.