What's the difference between “hidden” and “output” in PyTorch LSTM?

Question

I\'m having trouble understanding the documentation for PyTorch\'s LSTM module (and also RNN and GRU, which are similar). Regarding the outputs, it says:

Answer 1

It really depends on a model you use and how you will interpret the model. Output may be:

• a single LSTM cell hidden state
• several LSTM cell hidden states
• all the hidden states outputs

Output, is almost never interpreted directly. If the input is encoded there should be a softmax layer to decode the results.

Note: In language modeling hidden states are used to define the probability of the next word, p(w_t+1|w₁,...,w_t) =softmax(Wh_t+b).