LeCun Initialization

Before a network learns anything, its weights need a sensible starting scale. LeCun initialization gives each layer weights that are small enough to prevent runaway activations, but not so small that signals fade away as they pass through depth.

How the scale is chosen
For a layer with fan-in—the number of input connections feeding each neuron—LeCun initialization draws weights with variance:

Var(weight) = 1 / fan_in

Two common forms are:

LeCun normal: weights come from a normal distribution with standard deviation sqrt(1 / fan_in).
LeCun uniform: weights are sampled uniformly between -sqrt(3 / fan_in) and +sqrt(3 / fan_in).

The key idea is variance preservation. If inputs have a stable spread of values, summing many randomly weighted inputs could make that spread grow or shrink at every layer. Scaling by fan-in keeps a layer’s output variance near its input variance at the beginning of training.

Where it fits best
LeCun initialization is especially associated with SELU activations and self-normalizing neural networks. SELU is designed to pull activation statistics back toward a healthy mean and variance; LeCun’s scale gives that process the right starting conditions. In Keras, kernel_initializer="lecun_normal" is a common pairing for dense SELU layers. Biases are generally initialized to zero.

Why the choice matters
Bad initial scales create trouble before the optimiser has a chance to help: oversized weights can make activations and gradients explode, while undersized weights leave early layers receiving nearly zero gradient. LeCun initialization is not the universal default: for ReLU or GELU-heavy networks, He initialization better accounts for activations being suppressed on part of their range. Choosing the initializer to match the activation keeps the first training steps stable, making Adam or another optimiser adjust useful signals rather than recover from a poor starting point.

LeCun initialization sets each weight to a zero-mean random value with variance 1 / fan-in, where fan-in is the number of inputs to the neuron. It preserves activation variance through layers, particularly with self-normalizing or near-linear activations. This prevents signals from shrinking or growing excessively at the start of training, improving gradient flow and enabling stable optimization in deep networks.

Imagine setting the volume on every microphone before a concert. If they all start too quiet, nobody can hear the music. If they start too loud, everything becomes painful noise. LeCun initialization is a careful way to set the starting positions of a neural network’s many adjustable dials, called weights.

It gives the network a balanced starting point before it begins learning from examples. That helps information pass through the network at a sensible strength, rather than fading away or becoming overwhelming. The result is a steadier, more reliable start to training—like giving a learner a well-lit desk and a sharp pencil instead of asking them to begin in the dark.