Gradient Clipping

Training a neural network is like repeatedly nudging its parameters downhill on a loss landscape. Gradient clipping puts a safety limit on those nudges: when a computed gradient is dangerously large, it is reduced before the optimiser turns it into a parameter update.

What gets clipped

The usual method is global norm clipping. Treat every parameter gradient as one long vector, calculate its L2 norm, and, if that norm exceeds a threshold c, rescale every gradient by the same factor. Its direction is preserved, but its total size is capped:

if ||g|| > c:
    g = g * (c / ||g||)

Value clipping is another form: each individual gradient component is limited to a range such as −1 to +1. Norm clipping is generally preferred because it prevents an oversized update without distorting the gradient’s overall direction as sharply.

Why large gradients are a problem

During backpropagation, repeated multiplications through many layers or time steps can produce exploding gradients. One unusually large batch then causes an enormous update: weights jump into a bad region, the loss spikes, and training can become permanently unstable or turn into NaNs. This was especially important in recurrent networks, but it also protects deep transformers and other large models from rare unstable updates.

With PyTorch, torch.nn.utils.clip_grad_norm_ is called after loss.backward() and before optimizer.step().
In mixed-precision training, gradients must be unscaled before clipping; otherwise the artificial loss-scaling factor makes the threshold meaningless.
The threshold is a training control, not a cure for every problem. If clipping fires on nearly every step, a lower learning rate, better initialization, normalization, or a diagnosis of the loss may be needed.

Stability with a trade-off

Clipping makes a disastrous update bounded, which can rescue a run whose loss curve suddenly diverges after a few epochs. It adds negligible compute compared with backpropagation, but a threshold set too low suppresses useful learning and slows progress. Used as a guardrail rather than a substitute for sound optimisation settings, it lets the optimiser—such as Adam—keep making controlled steps even when a batch produces an abnormal gradient.

Gradient clipping limits gradient magnitude before an optimizer updates network parameters, typically by rescaling the full gradient vector when its norm exceeds a chosen threshold. It prevents rare, excessively large updates from causing exploding gradients, numerical instability, or parameter divergence. This is especially important in deep or recurrent networks, where unchecked gradients can destabilize training.

Imagine teaching someone to throw a ball. If they make a small mistake, you offer a small correction. But if they suddenly fling the ball wildly, you would not respond by shouting an enormous, confusing list of instructions. You would keep the correction to a sensible size.

Gradient clipping does this for a learning AI. During training, the network sometimes produces an unusually large correction signal. That can make its learning jump off course or become unstable. Gradient clipping puts a limit on how large that correction can be before the network updates itself. It helps training stay calm and steady, especially in models that process long sequences, such as text or speech.