Tensorflow warning: The graph couldn't be sorted in topological order?

Question

When I run tensorflow training (with custom defined graph, closed source), it outputs the warning:

2018-10-03 14:29:24.352895: E tensorfl

Answer 1

This error message can be encountered if there is a mismatch between your TF/CUDA version and the cudnn version that you're using. In this case, it seems to be a compatibility bug and doesn't necessarily indicate that you have bugs in your own code or model design. From what I can gather it happens between TF 1.10-1.14. Downgrading or upgrading TF (or cudnn) to the correct version should get rid of it. Note that while this is an error message, it is unclear if it actually produces any mistakes as models will train seemingly normally (except for the error message).