Understanding Variable scope example in Tensorflow

Question

I was looking at the mechanics section for Tensorflow, specifically on shared variables. In the section \"The problem\", they are dealing with a convolutional neural net, an

Answer 1

One has to create the variable set only once per whole training (and testing) set. The goal of variable scopes is to allow for modularization of subsets of parameters, such as those belonging to layers (e.g. when architecture of a layer is repeated, the same names can be used within each layer scope).

In your example you create parameters only in the model function. You can print out your variable names to see that it is assigned to the specified scope:

from __future__ import print_function

X = tf.placeholder("float") # create symbolic variables
Y = tf.placeholder("float")
print("X:", X.name)
print("Y:", Y.name)

def model(X):
    with tf.variable_scope("param"):
        w = tf.Variable(0.0, name="weights") # create a shared variable (like theano.shared) for the weight matrix
    print("w:", w.name)
    return tf.mul(X, w) 

The call to sess.run(train_op, feed_dict={X: x, Y: y}) only evaluates the value of train_op given the provided values of X and Y. No new variables (incl. parameters) are created there; therefore, it has no effect. You can make sure the variable names stay the same by again printing them out:

with tf.variable_scope("train"):
    print("X:", X.name)
    print("Y:", Y.name)
    for i in range(100):
        for (x, y) in zip(trX, trY):
            sess.run(train_op, feed_dict={X: x, Y: y})

You will see that variable names stay the same, as they are already initialized.

If you'd like to retrieve a variable using its scope, you need to use get_variable within a tf.variable_scope enclosure:

with tf.variable_scope("param"):
    w = tf.get_variable("weights", [1])
print("w:", w.name)