Stateful LSTM: When to reset states?

Question

Given X with dimensions (m samples, n sequences, and k features), and y labels with dimensions (m samples, 0/1):

Suppose I want t

Answer 1

If you use stateful=True, you would typically reset the state at the end of each epoch, or every couple of samples. If you want to reset the state after each sample, then this would be equivalent to just using stateful=False.

Regarding the loops you provided:

for e in epoch:
    for m in X.shape[0]:          #for each sample
        for n in X.shape[1]:      #for each sequence

note that the dimension of X are not exactly

 (m samples, n sequences, k features)

The dimension is actually

(batch size, number of timesteps, number of features)

Hence, you are not supposed to have the inner loop:

for n in X.shape[1]

Now, regarding the loop

for m in X.shape[0]

since the enumeration over batches is done in keras automatically, you don't have to implement this loop as well (unless you want to reset the states every couple of samples). So if you want to reset only at the end of each epoch, you need only the external loop.

Here is an example of such architecture (taken from this blog post):

batch_size = 1
model = Sequential()
model.add(LSTM(16, batch_input_shape=(batch_size, X.shape[1], X.shape[2]), stateful=True))
model.add(Dense(y.shape[1], activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
for i in range(300):
    model.fit(X, y, epochs=1, batch_size=batch_size, verbose=2, shuffle=False)
    model.reset_states()