He Initialization

Before a network learns anything, its weights must start at a sensible scale. He initialization, also called Kaiming initialization, gives layers using ReLU-like activations a starting point where signals can travel through depth without immediately shrinking away or blowing up.

A neuron computes a weighted sum of its inputs, then applies an activation such as ReLU. ReLU replaces every negative value with zero, so roughly half the incoming signal is discarded at initialization. If weights used the same scale as an initialization designed for symmetric activations such as tanh, each layer would lose too much variance. In a deep stack, activations would fade toward zero and early layers would receive tiny gradients.

He initialization compensates by drawing each weight with variance:

Var(W) = 2 / fan_in

Here, fan_in is the number of inputs feeding one neuron (or, for a convolution, input channels times kernel area). A normal version draws weights from N(0, 2/fan_in); the uniform version uses limits of ±sqrt(6/fan_in). Biases are commonly initialized to zero.

The factor of 2 keeps the typical magnitude of activations roughly stable from layer to layer despite ReLU’s zeroing behavior. This gives backpropagation a healthier path too: gradients reaching early layers remain large enough to update their weights. For a leaky ReLU with negative slope a, the variance becomes 2 / ((1 + a²) fan_in), reflecting that negative values are no longer completely removed.

In PyTorch, torch.nn.Linear and convolution layers use Kaiming-style initialization by default, while torch.nn.init.kaiming_normal_ exposes it directly. It adds no training-time memory or compute cost: it is only the network’s starting condition. A poor scale can make a loss curve stall immediately from vanishing activations, or spike and diverge from exploding values. He initialization does not replace good learning rates, normalization, or residual connections, but it ensures a ReLU-based network begins training on stable ground.