Exploding Gradients

Training a deep network means sending an error signal backward so each weight can be adjusted. With exploding gradients, that signal grows uncontrollably as it travels through layers or time steps, turning a sensible correction into a huge, destabilising update.

How the explosion happens

Backpropagation applies the chain rule: a gradient is repeatedly multiplied by each layer’s derivatives and weight matrices. When these factors are consistently larger than one, their product can grow exponentially with depth. A gradient of 1.2 multiplied across 100 steps is enormous. This was especially severe in older recurrent neural networks, where the same recurrent weights are reused across many time steps, but it can affect any very deep network.

What it looks like during training

Once gradients explode, an optimiser such as Adam or SGD takes oversized parameter steps. The loss may suddenly spike after several stable epochs, predictions become erratic, and weights or activations can reach NaN or infinity. Common warning signs include:

Gradient norms jumping from ordinary values to thousands or millions.
A loss curve that abruptly diverges rather than gradually improving.
Numerical errors, especially with mixed-precision training.

How networks control it

Gradient clipping is the direct safety brake: if a gradient’s norm exceeds a threshold, training rescales it while preserving its direction. PyTorch’s torch.nn.utils.clip_grad_norm_ is widely used for this purpose. Stable initialization, a lower learning rate, LayerNorm, and architectures with short gradient paths—such as ResNet skip connections or gated LSTM/GRU cells—reduce the underlying amplification. Clipping does not repair a poorly designed model by itself, but it prevents one disastrous update from destroying a run and gives the optimiser a stable path to continue learning.

Exploding gradients occur when backpropagated derivatives grow exponentially across layers or time steps, producing extremely large parameter updates. This makes optimization unstable: weights can overflow, the loss can diverge, and training can fail abruptly. Gradient clipping, careful initialization, normalization, and architectures with controlled gradient flow help prevent this instability.

Imagine trying to adjust a huge mixing desk, where each wrong note makes you turn every dial more and more wildly. Soon the sound becomes deafening and impossible to control. Exploding gradients are a similar problem when an AI is learning.

The network learns by making many tiny corrections after seeing an example. But sometimes those correction signals grow far too large as they travel through the network. The AI then makes enormous changes instead of careful ones, so its performance can suddenly become unstable or collapse.

It matters because learning needs steady, useful feedback—not panicked overcorrections. Designers use safeguards to keep these learning signals at a manageable size.