Xavier (Glorot) Initialization

Before a network learns anything, its weights need a sensible starting point. Xavier (Glorot) initialization chooses small random values that let information flow through many layers without immediately fading into nothing or blowing up.

Each neuron combines inputs using its weights. If those weights begin too large, activations grow as they pass through layers; if too small, they shrink. Backpropagation has the same problem in reverse: gradients can explode or vanish before reaching early layers. Xavier initialization balances these two directions by scaling the random weights according to a layer’s number of input connections (fan-in) and output connections (fan-out).

• For a uniform distribution: W ~ U(-a, a), where a = √(6 / (fan-in + fan-out)).
• For a normal distribution: Var(W) = 2 / (fan-in + fan-out).

Glorot initialization was designed especially for roughly zero-centered activations such as tanh. It aims to preserve the variance of both activations and gradients from one layer to the next. That does not mean every layer produces identical values; it means the starting scale is stable enough for learning to begin. For networks dominated by ReLU-style activations, He (Kaiming) initialization is usually a better match because ReLU discards negative values and changes the variance differently.

A deep multilayer perceptron initialized with unscaled random weights can show a flat loss curve because early layers receive almost zero gradient, or it can diverge within a few updates as activations become enormous. Xavier initialization removes this avoidable instability at the start of training. In PyTorch, torch.nn.init.xavier_uniform_ and xavier_normal_ implement it directly. It costs essentially nothing beyond sampling the initial weights, but it does not replace good learning rates, normalization, or architecture choices: it gives optimization a stable launch point rather than guaranteeing a successful landing.