Leaky ReLU

Leaky ReLU is a small adjustment to the familiar ReLU activation that keeps a neuron from becoming completely silent. It gives negative inputs a narrow “escape route,” so learning signals can still travel through them during training.

How it works
For an input value x, Leaky ReLU returns x when x is positive, and returns a small fraction of x when it is negative:

LeakyReLU(x) = x        if x > 0
             = αx       if x ≤ 0

Here, α is a fixed small positive slope, commonly 0.01. Standard ReLU outputs exactly zero for every negative input, and its gradient there is also zero. If a neuron stays on that negative side, gradient descent cannot adjust its incoming weights: this is the dying ReLU problem. Leaky ReLU instead supplies a gradient of α on the negative side, allowing that neuron to recover.

Why this changes training
In a deep network, backpropagation multiplies gradients through many layers. A zero gradient from inactive ReLUs can stop updates entirely for particular units, especially after an overly large learning-rate step shifts their inputs negative. Leaky ReLU does not solve every vanishing-gradient problem, but it prevents this specific hard cutoff. Its piecewise-linear shape also remains cheap to compute and avoids the saturation of sigmoid or tanh activations.

Practical use and trade-offs
Leaky ReLU can replace ReLU in a hidden layer directly; in PyTorch, this is torch.nn.LeakyReLU(negative_slope=0.01). It behaves identically during training and inference, unlike dropout. The cost is that negative activations are no longer exactly zero, so the representation is less sparse. Setting α too large also weakens the contrast between positive and negative signals. A related layer, PReLU, learns α from data rather than fixing it.

Leaky ReLU is a piecewise-linear activation function that outputs x for positive inputs and a small fixed fraction, αx, for negative inputs, where α is typically small (for example, 0.01). Unlike standard ReLU, it preserves a nonzero negative-side gradient, reducing permanently inactive (“dying”) neurons and supporting more reliable gradient flow during training.

Imagine a row of light switches in a learning machine. A standard switch is fully off for certain signals, so nothing gets through. Sometimes, though, a switch can stay off for so long that it never becomes useful again.

Leaky ReLU is like a switch that never shuts completely. For positive signals, it lets information pass normally. For negative signals, it lets through a tiny trickle instead of blocking them entirely. This small “leak” helps a neural network keep learning from more of its experience, rather than leaving some of its internal units permanently inactive. It is a simple safeguard that can make training more reliable.