Hidden State

A hidden state is a neural network’s working memory while it processes a sequence. At each step, it carries forward a compact record of what the network has seen so far, so the current output can depend on earlier inputs rather than treating every item in isolation.

How the state is updated
In a basic recurrent neural network, the hidden state at time t is computed from two ingredients: the current input and the previous hidden state. Conceptually:

h_t = activation(W_x x_t + W_h h_(t-1) + b)

Here, h_t is the new hidden state, x_t is the current input, and the learned weights decide what information to retain, change, or discard. The state is “hidden” because it is an internal vector, not necessarily the model’s visible prediction. A network might read one token at a time, update its state after each token, and use the final state to make a classification.

Why LSTMs and GRUs changed the design
A plain RNN repeatedly transforms its state, which makes gradients shrink or explode during backpropagation through time. When gradients vanish, information from early steps cannot influence learning: the model effectively forgets the beginning of a long sequence. LSTM and GRU layers use gates to control this memory flow:

Forget gates remove information that is no longer useful.
Input gates decide what new information enters memory.
Output gates control which stored information becomes the visible hidden state.

Training and practical use
In an LSTM, the durable cell state and exposed hidden state play related but distinct roles; frameworks such as PyTorch return both. Hidden states must be reset between unrelated sequences, or detached when processing very long streams in chunks, otherwise gradients and memory usage grow uncontrollably. Carrying a state across appropriate chunks preserves context; carrying it across unrelated examples leaks information and corrupts training.

Hidden state is a learned internal vector that carries information from earlier sequence steps to later ones in a recurrent neural network. At each step, the network updates it using the current input and previous hidden state. It provides the network’s working memory, enabling predictions to depend on prior context; without an effective hidden state, recurrent models cannot represent temporal dependencies.

Think of someone reading a story while keeping a few important details in mind: who the characters are, what just happened, and what might matter next. A recurrent neural network uses a hidden state in a similar way.

The hidden state is the network’s small, temporary “memory” as it moves through a sequence, such as words in a sentence, notes in music, or measurements over time. It carries forward useful context from earlier steps, helping the network interpret what comes next. For example, in “The dog chased the ball because it…,” this memory helps the system understand what “it” likely refers to.