Tensorflow LSTM Regularization

Question

I was wondering how one can implement l1 or l2 regularization within an LSTM in TensorFlow? TF doesn\'t give you access to the internal weights of the LSTM, so I\'m not certain

Answer 1

TL;DR; Save all the parameters in a list, and add their L^n norm to the objective function before making gradient for optimisation

1) In the function where you define the inference

net = [v for v in tf.trainable_variables()]
return *, net

2) Add the L^n norm in the cost and calculate the gradient from the cost

weight_reg = tf.add_n([0.001 * tf.nn.l2_loss(var) for var in net]) #L2

cost = Your original objective w/o regulariser + weight_reg

param_gradients = tf.gradients(cost, net)

optimiser = tf.train.AdamOptimizer(0.001).apply_gradients(zip(param_gradients, net))

3) Run the optimiser when you want via

_ = sess.run(optimiser, feed_dict={input_var: data})