ReLU
ReLU is a tiny operation with a large effect: it decides whether a neuron passes its signal onward or shuts it off. Its simplicity helped make very deep neural networks practical to train.
How it works
ReLU means Rectified Linear Unit. Given a neuron’s input value x, it returns max(0, x): negative values become zero, while positive values pass through unchanged. A network needs this kind of nonlinearity because stacking layers that are only linear is equivalent to one larger linear layer; depth would add no expressive power.
Why gradients flow better
For a positive input, ReLU’s derivative is 1. During backpropagation, the gradient can therefore pass through that active neuron without being repeatedly shrunk by the activation itself. This contrasts with sigmoid or tanh, whose derivatives become tiny in saturated regions and can leave early layers learning extremely slowly. At exactly zero ReLU has no unique derivative; frameworks such as PyTorch conventionally use zero.
Benefits and failure modes
ReLU is cheap to compute and produces many exact zeros, creating sparse activations. It is commonly paired with He initialization, which is designed to keep signal scale stable through ReLU layers. Its key weakness is the dying ReLU: if a neuron’s inputs stay negative, its output and gradient are both zero, so it receives no update and can remain permanently inactive. An excessively large learning rate can trigger this across many units, visible as stalled loss improvement. Leaky ReLU addresses this by retaining a small negative-side slope. In a ResNet block or ordinary PyTorch nn.ReLU layer, ReLU is usually placed after a linear/convolution operation and, depending on the block design, around normalization and skip connections.
ReLU (Rectified Linear Unit) is an activation function defined as max(0, x): it outputs zero for negative inputs and the input itself for positive inputs. Its simple, non-saturating positive region supports efficient gradient propagation and computationally cheap training in deep networks. However, units receiving persistently negative inputs output zero and stop updating, known as the dying ReLU problem.
Think of a room full of tiny decision-makers. Each one receives a signal and asks a simple question: “Is this useful enough to pass on?” ReLU, short for “rectified linear unit,” is the rule it uses. Positive signals are passed forward; negative signals are treated as zero.
This simple filter helps a neural network focus on features that matter—perhaps an edge in a photo, a spoken sound, or a useful word pattern. By letting strong positive clues continue while silencing unhelpful ones, ReLU helps deep AI systems learn clearer, more flexible patterns. It is popular because it is simple, fast, and often makes learning in many-layered networks much more practical.