Tensorflow warning: The graph couldn't be sorted in topological order?

Question

When I run tensorflow training (with custom defined graph, closed source), it outputs the warning:

2018-10-03 14:29:24.352895: E tensorfl

Answer 1

This error message can be encountered if there is a mismatch between your TF/CUDA version and the cudnn version that you're using. In this case, it seems to be a compatibility bug and doesn't necessarily indicate that you have bugs in your own code or model design. From what I can gather it happens between TF 1.10-1.14. Downgrading or upgrading TF (or cudnn) to the correct version should get rid of it. Note that while this is an error message, it is unclear if it actually produces any mistakes as models will train seemingly normally (except for the error message).

Answer 2

A topological ordering of a directed graph is an ordering of its vertices in such a way that whenever there is an edge from vertex u to vertex v, vertex u comes before vertex v in the ordering.

This kind of ordering is possible for every directed acyclic graph but not for arbitrary graphs. There is most likely some kind of cycle in your graph that prevents the sorting algorithm from succeeding. So, the way to go is searching for the cycle and removing it in some way.

As an example, consider a very small graph with two vertices, u and v, and two edges, u -> v and v -> u. There is no way to sort u and v in accordance with the requirements given above.

Without further information on your code it is hard to say exactly what is going on.