Vanishing Gradients

Training a deep network is like sending a correction signal backward through every layer. With vanishing gradients, that signal fades as it travels, so the layers closest to the input receive almost no instruction about how to improve.

Why the signal fades
Backpropagation uses the chain rule: a layer’s gradient is formed by multiplying gradients and derivatives from all the layers after it. When many of those factors are smaller than 1, their product shrinks rapidly. For example, multiplying 0.5 by itself 20 times produces roughly one millionth. The early layers then get parameter updates so tiny that, at the chosen learning rate, they barely move.

Common causes and visible symptoms
The problem is especially severe with saturating activation functions such as sigmoid and tanh. In their flat outer regions, their derivatives approach zero, weakening the backward signal. Poor weight initialization can push activations into these regions from the first batch. A loss curve may fall briefly and then plateau, while inspection shows gradients in early layers are far smaller than those near the output. Deep recurrent networks face the same issue across time steps: information from far earlier positions cannot meaningfully influence learning.

How modern networks preserve gradient flow

ReLU-family activations provide a stronger derivative on their active side than sigmoid.
Xavier or He initialization keeps signal scales sensible at the start of training.
Residual connections, used in ResNets and transformer blocks, create short routes for gradients: a block learns a change to its input rather than forcing all information through many transformations.
LayerNorm helps keep activations in stable ranges; transformer blocks commonly place it around residual pathways.

These choices do not merely make optimization faster. They make depth usable: without a reliable backward signal, adding layers adds parameters but not learnable capability.

Vanishing gradients occur when backpropagated derivatives shrink toward zero across many layers or time steps, leaving early parameters with negligible updates. This prevents lower layers from learning useful representations and can stall training in deep networks. Non-saturating activations, careful initialization, normalization, residual connections, and gated recurrent units help preserve gradient flow.

Imagine a long game of “telephone,” where a message is whispered from person to person. By the time it reaches the first person, it may be so faint or altered that it is no longer useful. Vanishing gradients are a similar problem in deep learning.

When a neural network learns, it sends feedback backward through its layers, telling each part how much it should improve. In a very deep network, that feedback can shrink as it travels backward. Early layers then receive almost no useful signal, so they barely learn at all.

This matters because those early layers often need to spot basic patterns, such as edges in an image or simple word features in text. Designs that preserve stronger feedback help deep networks learn effectively.