How does one debug NaN values in TensorFlow?

Question

I was running TensorFlow and I happen to have something yielding a NaN. I\'d like to know what it is but I do not know how to do this. The main issue is that in a \"normal\"

Answer 1

Current implementation of tfdbg.has_inf_or_nan seems do not break immediately on hitting any tensor containing NaN. When it does stop, the huge list of tensors displayed are not sorted in order of its execution. A possible hack to find the first appearance of Nans is to dump all tensors to a temporary directory and inspect afterwards. Here is a quick-and-dirty example to do that. (Assuming the NaNs appear in the first few runs)