Deep-Learning Nan loss reasons

Question

Perhaps too general a question, but can anyone explain what would cause a Convolutional Neural Network to diverge?

Specifics:

I am using Tensorflow\'s iris_tra

Answer 1

I'd like to plug in some (shallow) reasons I have experienced as follows:

1. we may have updated our dictionary(for NLP tasks) but the model and the prepared data used a different one.
2. we may have reprocessed our data(binary tf_record) but we loaded the old model. The reprocessed data may conflict with the previous one.
3. we may should train the model from scratch but we forgot to delete the checkpoints and the model loaded the latest parameters automatically.

Hope that helps.