Tensorflow LSTM - Matrix multiplication on LSTM cell

Question

I'm making a LSTM neural network in Tensorflow.

The input tensor size is 92.

import tensorflow as tf
from tensorflow.contrib import rnn
import data

test_x, train_x, test_y, train_y = data.get()

# Parameters
learning_rate = 0.001
epochs = 100
batch_size = 64
display_step = 10

# Network Parameters
n_input = 28   # input size
n_hidden = 128 # number of hidden layers
n_classes = 20 # output size

# Placeholders
x = tf.placeholder(dtype=tf.float32, shape=[None, n_input])
y = tf.placeholder(dtype=tf.float32, shape=[None, n_classes])

# Network
def LSTM(x):
    W = tf.Variable(tf.random_normal([n_hidden, n_classes]), dtype=tf.float32) # weights
    b = tf.Variable(tf.random_normal([n_classes]), dtype=tf.float32) # biases

    x_shape = 92

    x = tf.transpose(x)
    x = tf.reshape(x, [-1, n_input])
    x = tf.split(x, x_shape)

    lstm = rnn.BasicLSTMCell(
        num_units=n_hidden,
        forget_bias=1.0
    )
    outputs, states = rnn.static_rnn(
        cell=lstm,
        inputs=x,
        dtype=tf.float32
    )

    output = tf.matmul( outputs[-1], W ) + b

    return output

# Train Network
def train(x):
    prediction = LSTM(x)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        output = sess.run(prediction, feed_dict={"x": train_x})
        print(output)

train(x)

I'm not getting any errors, but I'm feeding an input tensor of size 92, and the matrix multiplication in the LSTM function returns a list containing one result vector, when the desired amount is 92, one result vector per input.

Is the problem that I'm matrix multiplying only the last item in the outputs array? Like this:

output = tf.matmul( outputs[-1], W ) + b

instead of:

output = tf.matmul( outputs, W ) + b

This is the error I get when I do the latter:

ValueError: Shape must be rank 2 but is rank 3 for 'MatMul' (op: 'MatMul') with input shapes: [92,?,128], [128,20].

Answer 1

static_rnn for making the simplest recurrent neural net.Here's the tf documentation.So the input to it should be a sequence of tensors. Let's say you want to input 4 words calling "Hi","how","Are","you". So your input place holder should consist of four n(size of each input vector) dimensional vectors corresponding to each words.

I think there's something wrong with your place holder. You should initialize it with number of inputs to the RNN. 28 is number of dimensions in each vector. I believe 92 is the length of the sequence. (more like 92 lstm cell)

In the output list you will get set of vectors equal to length of sequence each of size equal to number of hidden units.