Keras LSTM - feed sequence data with Tensorflow dataset API from the generator

Question

I am trying to solve how I can feed data to my LSTM model for training. (I will simplify the problem in my example below.) I have the following data format in csv files in my dataset.

Timestep    Feature1    Feature2    Feature3    Feature4    Output
1           1           2           3           4           a
2           5           6           7           8           b
3           9           10          11          12          c 
4           13          14          15          16          d
5           17          18          19          20          e
6           21          22          23          24          f
7           25          26          27          28          g
8           29          30          31          32          h
9           33          34          35          36          i
10          37          38          39          40          j

The task is to estimate the Output of any future timestep based on the data from last 3 timesteps. Some input-output exapmles are as following:

Example 1: Input:

Timestep    Feature1    Feature2    Feature3    Feature4    
1           1           2           3           4           
2           5           6           7           8           
3           9           10          11          12           

Output: c

Example 2: Input:

Timestep    Feature1    Feature2    Feature3    Feature4    
2           5           6           7           8           
3           9           10          11          12           
4           13          14          15          16          

Output: d

Example 3: Input:

Timestep    Feature1    Feature2    Feature3    Feature4   
3           9           10          11          12          
4           13          14          15          16         
5           17          18          19          20         

Output: e

And when feeding the data to the model, I would like to shuffle the data in a way so that I do not feed consecutive sequences when training. With other words, I ideally would like to feed the data sequences like timesteps 3,4,5 in one step, maybe timesteps 5,6,7 in the next step, and maybe 2,3,4 in the following step, and so on.. And I preferably do not want to feed the data as 1,2,3 first, then 2,3,4, then 3,4,5, and so on...

When training my LSTM network, I am using Keras with Tensorflow backend. I would like to use a generator when feeding my data to the fit_generator(...) function.

My desire is to use Tensorflow's dataset API to fetch the data from csv files. But I could not figure out how to make the generator return what I need. If I shuffle the data with Tensorflow's dataset API, it will destroy the order of the timesteps. The generator should also return batches that include multiple sequence examples. For instance, if the batch size is 2, then it may need to return 2 sequences like timesteps 2,3,4 and timesteps 6,7,8.

Hoping that I could explain my problem... Is it possible to use Tensorflow's dataset API in a generator function for such a sequence problem so that I can feed batches of sequences as I explained above? (The generator needs to return data with the shape [batch_size, length_of_each_sequence, nr_inputs_in_each_timestep], where length_of_each_sequence=3 and nr_of_inputs_in_each_timestep=4 in my example.) Or is the best way to do this to write a generator in Python only, maybe by using Pandas..?

ADDENDUM 1:

I have done the following experiment after seeing the answer from @kvish.

import tensorflow as tf
import numpy as np
from tensorflow.contrib.data.python.ops import sliding

sequence = np.array([ [[1]], [[2]], [[3]], [[4]], [[5]], [[6]], [[7]], [[8]], [[9]] ])
labels = [1,0,1,0,1,0,1,0,1]

# create TensorFlow Dataset object
data = tf.data.Dataset.from_tensor_slices((sequence, labels))

# sliding window batch
window_size = 3
window_shift = 1
data = data.apply(sliding.sliding_window_batch(window_size=window_size, window_shift=window_shift))
data = data.shuffle(1000, reshuffle_each_iteration=False)
data = data.batch(3)

#iter = dataset.make_initializable_iterator()
iter = tf.data.Iterator.from_structure(data.output_types, data.output_shapes)
el = iter.get_next()

# create initialization ops 
init_op = iter.make_initializer(data)

NR_EPOCHS = 2
with tf.Session() as sess:
    for e in range (NR_EPOCHS):
      print("\nepoch: ", e, "\n")
      sess.run(init_op)
      print("1  ", sess.run(el))
      print("2  ", sess.run(el))
      print("3  ", sess.run(el))

And here is the output:

epoch:  0 

1   (array([[[[6]],[[7]],[[8]]],  [[[1]],[[2]],[[3]]],  [[[2]],[[3]],[[4]]]]), 
     array([[0, 1, 0],  [1, 0, 1],  [0, 1, 0]], dtype=int32))

2   (array([[[[7]],[[8]],[[9]]],  [[[3]],[[4]],[[5]]],  [[[4]],[[5]],[[6]]]]), 
     array([[1, 0, 1],  [1, 0, 1],  [0, 1, 0]], dtype=int32))

3   (array([[[[5]],[[6]],[[7]]]]), array([[1, 0, 1]], dtype=int32))

epoch:  1 

1   (array([[[[2]],[[3]],[[4]]],  [[[7]],[[8]],[[9]]],  [[[1]],[[2]],[[3]]]]), 
     array([[0, 1, 0],  [1, 0, 1],  [1, 0, 1]], dtype=int32))

2   (array([[[[5]],[[6]],[[7]]],  [[[3]],[[4]],[[5]]],  [[[4]],[[5]],[[6]]]]), 
     array([[1, 0, 1],  [1, 0, 1],  [0, 1, 0]], dtype=int32))

3   (array([[[[6]],[[7]],[[8]]]]), 
     array([[0, 1, 0]], dtype=int32))

I could not try it on csv file reading yet but I think that this approach should be working quite fine!

But as I see it, the reshuffle_each_iteration parameter is making no difference. Is this really needed? Results are not necessarily identical when it is set to True or False. What is this reshuffle_each_iteration parameter supposed to do here?

Answer 1

I think this answer might be close to what you are looking for!

You create batches by sliding over windows, and then shuffle the input in your case. The shuffle function of the dataset api has a reshuffle_after_each_iteration parameter, which you might probably want to set to False if you want to experiment with setting a random seed and looking at the order of shuffled outputs.