How Session.run() chooses which sub-graph to run

问题

I was trying to understand how Session.run() works in Tensorflow flow. I know that Session.run() runs the sub-graph specified by the "fetch" argument we give it. Since depending on which part of the sub-graph is executed first we might get different results, I was trying to see if that is really the case. Suppose we compare the output of this code:

import tensorflow as tf

x = tf.Variable(42)
assign1 = tf.assign(x, 13)
assign2 = tf.assign(x, 14)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1, val2, val3 = sess.run([x, assign1, assign2])
    print(val1, val2, val3)

with this code:

import tensorflow as tf

x = tf.Variable(42)
assign2 = tf.assign(x, 14)
assign1 = tf.assign(x, 13)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1, val2, val3 = sess.run([x, assign1, assign2])
    print(val1, val2, val3)

(The only change is assigning of 14 comes first in the second code). I was expecting to see that the first code produces output 14, 14, 14 while the second one produced 13, 13, 13.

However, the result is that the first produced 13, 13, 13 while the second on produced 14, 14, 14.

Why does this happen?

Update: Following Chosen Answer:

I don't quite get what is independent and what is not. For example, in the following code:

x = tf.Variable([1, 2, 3, 4, 5])

def foo():
    tmp_list = []
    assign = tf.assign(x[4], 100)
    for i in range(0, 5):
        tmp_list.append(x[i])
    return tmp_list

z = foo()

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    val1, val2 = sess.run([z, assign])
    print(val1, val2)

In my Session.run() I have both z and assign. To get z, I need foo(). When foo() is ran, assign = tf.assign(x[4], 100) is also ran. However, the output is [1, 2, 3, 4, 5] 13 and I have no idea where the 13 comes from.

回答1:

There is no guarantee which of independent ops is performed first, so you can get 13 or 14 in both cases. Your result is pure luck. Try this:

with tf.Session() as sess:
  sess.run(tf.global_variables_initializer())
  for i in range(100):
    val1, val2, val3 = sess.run([x, assign1, assign2])
    print(val1, val2, val3)

... and you'll see 13 and 14 printed out.

Update for the subsequent question:

z = foo() (which is just list of x slices) does not depend on assign. Note that foo is called just once before the session is started. I think, the best to see it is to visualize the graph in tensorboard.

If you say...

sess.run(z)

... tensorflow won't run assign, because it doesn't run a python function, it evaluates tensors. And tensor x or x[i] does not depend on the op assign.

Dependency is defined by operations that are performed with tensors. So if you had...