How to expand a Tensorflow Variable

Question

Is there any way to make a Tensorflow Variable larger? Like, let\'s say I wanted to add a neuron to a layer of a neural network in the middle of training. How would I go a

Answer 1

There are various ways you could accomplish this.

1) The second answer in that post (https://stackoverflow.com/a/33662680/5548115) explains how you can change the shape of a variable by calling 'assign' with validate_shape=False. For example, you could do something like

# Assume var is [m, n] 
# Add the new 'data' of shape [1, n] with new values
new_neuron = tf.constant(...)  

# If concatenating to add a row, concat on the first dimension.
# If new_neuron was [m, 1], you would concat on the second dimension.
new_variable_data = tf.concat(0, [var, new_neuron])  # [m+1, n]

resize_var = tf.assign(var, new_variable_data, validate_shape=False)

Then when you run resize_var, the data pointed to by 'var' will now have the updated data.

2) You could also create a large initial variable, and call tf.slice on different regions of the variable as training progresses, since you can dynamically change the 'begin' and 'size' attributes of slice.