问题
I have some data that is sampled at at a very high rate (on the order of hundreds of times per second). This results in a sequence length that is huge (~90,000 samples) on average for any given instance. This entire sequence has a single label. I am trying to use an LSTM neural network to classify new sequences as one of these labels (multiclass classification).
However, using an LSTM with a such a large sequence length results in a network that is quite large.
What are some methods to effectively 'chunk' these sequences so that I could reduce the sequence length of the neural network, yet still maintain the information captured in the entire instance?
回答1:
When you have very long sequences RNNs can face the problem of vanishing gradients and exploding gradients.
There are methods. The first thing you need to understand is why we need to try above methods? It's because back propagation through time can get real hard due to above mentioned problems.
Yes introduction of LSTM has reduced this by very large margin but still when it's is so long you can face such problems.
So one way is clipping the gradients. That means you set an upper bound to gradients. Refer to this stackoverflow question
Then this problem you asked
What are some methods to effectively 'chunk' these sequences?
One way is truncated back propagation through time. There are number of ways to implement this truncated BPTT. Simple idea is
- Calculate the gradients only for number of given time steps That means if your sequence is 200 time steps and you only give 10 time steps it will only calculate gradient for 10 time step and then pass the stored memory value in that 10 time step to next sequence(as the initial cell state) . This method is what tensorflow using to calculate truncated BPTT.
2.Take the full sequence and only back propagate gradients for some given time steps from selected time block. It's a continuous way
Here is the best article I found which explains these trunacated BPTT methods. Very easy. Refer to this Styles of Truncated Backpropagation
来源：https://stackoverflow.com/questions/44478272/how-to-handle-extremely-long-lstm-sequence-length