Backpropagation Through Time (BPTT)
A recurrent network does not process an input just once: it carries a hidden state forward, letting earlier steps influence later ones. Backpropagation Through Time (BPTT) is how the network learns which parts of that carried information helped or hurt its final prediction.
Unrolling the sequence
BPTT treats one recurrent network used repeatedly across time as though it were a long chain of copies. At each step, the network reads an input, updates its hidden state, and produces an output when needed. The copies all share the same weights: there is still only one set of recurrent parameters to learn. During the backward pass, the loss gradient travels from later steps back through every earlier hidden-state update, using the chain rule. Each shared weight receives the sum of gradient contributions from every time step where it was used.
Why long dependencies are difficult
The gradient passing backward is repeatedly multiplied by derivatives and recurrent weight matrices. Across many steps, those products can create two classic failures:
- Vanishing gradients: the signal shrinks toward zero, so early events receive almost no learning signal.
- Exploding gradients: the signal grows uncontrollably, producing unstable updates, sudden loss spikes, or numerical errors.
For example, a model asked to use information from 100 steps earlier can fail because the gradient reaching that early state has effectively disappeared. LSTMs and GRUs use gated pathways to preserve information and gradient flow; gradient clipping limits destructive explosions.
Training in practice
Full BPTT stores the activations from every step, making memory and computation scale with sequence length. Training commonly uses truncated BPTT: after, say, 32 or 128 steps, the hidden state is carried forward but detached from its earlier computation graph. This is practical for long streams, but it prevents learning dependencies beyond that backward window. In PyTorch, this detachment is typically done with hidden = hidden.detach(). A window that is too short creates a model that learns local patterns yet misses delayed causes; an unbounded window can exhaust memory or destabilize training.
Backpropagation Through Time (BPTT) trains recurrent neural networks by unrolling their repeated computation across sequence steps and applying backpropagation through the resulting time-ordered graph. It computes gradients for shared parameters from their effects at every step. BPTT enables sequence learning, but long unrolled paths can cause vanishing or exploding gradients, limiting how far dependencies can be learned.
Imagine teaching someone to play a song by reviewing the whole performance afterward: “That wrong note near the end may have started with how you began the rhythm.” Backpropagation Through Time, often called BPTT, is how an AI with a memory of earlier steps learns in a similar way.
It is used for tasks that unfold in sequence, such as reading a sentence, predicting the next word, or tracking speech. After the AI makes a prediction, BPTT looks back through the earlier moments that influenced it. It helps the network adjust its habits so future predictions better reflect what came before.