Truncated BPTT

Training a recurrent network means teaching it not only what to do at the current step, but how an earlier step influenced what came later. Truncated backpropagation through time (TBPTT) makes that teaching job manageable by limiting how far backward through a sequence the network looks during each update.

What is truncated

An RNN, GRU, or LSTM reuses its hidden state across time: the state at step t helps produce the state at t+1. Full backpropagation through time builds a gradient path through every step in an entire sequence. For a very long stream, that requires storing every intermediate activation, which is expensive in memory and compute.

With TBPTT, training processes a fixed window—for example, 32 steps—then computes gradients only through that window. The hidden state is carried numerically into the next window, preserving forward-pass context, but it is detached from the computation graph. In PyTorch, this is commonly expressed with hidden = hidden.detach(). The next update therefore cannot assign credit or blame to events before the window boundary.

The practical trade-off

• Longer windows capture longer dependencies, but consume more memory and take longer per update.
• Shorter windows make training feasible on long or continuous sequences, but prevent learning relationships beyond the truncation length.
• Windows can be consecutive or overlap. Overlap gives more positions access to nearby context, at additional compute cost.

For example, an LSTM trained on a continuous sensor stream can carry its state indefinitely while updating every 64 readings. It can react to a pattern established before the current chunk, but its parameters receive gradient evidence only from the most recent 64 steps. If the task depends on a cue 500 steps earlier, a 64-step truncation can leave learning stalled even when the forward state contains that cue.

Why it matters for stable training

TBPTT is primarily a resource-control technique, not a cure for vanishing or exploding gradients. It makes recurrent training fit in memory and permits frequent optimizer updates, but the chosen length changes the problem the model can learn. Too short creates artificial amnesia; too long can exhaust memory and expose unstable long gradient chains. Gated architectures such as LSTMs and GRUs, gradient clipping, and a sensible learning rate help make the remaining gradient path trainable.