Random Initialization
Before a neural network can learn, it needs a starting point. Random initialization gives each trainable weight a small, different value, so neurons begin with slightly different responses and can develop different jobs during training.
Breaking symmetry
If every weight in a layer started at zero, or at the same constant, every neuron would produce the same output, receive the same gradient, and remain identical after every update. A layer with 100 identical neurons would effectively behave like one neuron. Random values break this symmetry: each neuron sees the input through a different initial set of weights, so gradient descent can shape them into useful, distinct features. Biases are commonly initialized to zero because weights already provide the needed asymmetry.
The scale is as important as randomness
Random does not mean “pick arbitrary numbers.” In a deep network, activations and gradients pass through many layers. If initial weights are too large, repeated multiplications make values explode; if too small, they fade toward zero. Both cases stop useful learning. Modern schemes choose a random distribution whose variance depends on a layer’s number of input and output connections:
- Xavier/Glorot initialization is designed for layers using balanced activations such as tanh.
- He/Kaiming initialization uses a larger variance suited to ReLU-family activations, which discard negative values.
What failure looks like
Poor initialization can make the loss become NaN within a few updates, remain flat because early-layer gradients vanish, or force an optimizer such as Adam to spend many epochs recovering from a bad starting scale. Good initialization does not solve every training problem—learning rate, normalization, and architecture still matter—but it gives signals and gradients a stable first path through the network. In a ResNet, skip connections further protect that path, making very deep models far less fragile at initialization.
Random initialization assigns each trainable weight a randomly sampled value before learning begins, breaking symmetry so neurons develop different features. The distribution and scale are chosen to preserve reasonable activation and gradient variance across layers, as in Xavier or He initialization. It matters because identical or poorly scaled starting weights prevent effective specialization and can cause vanishing or exploding signals, slowing or destabilizing training.
Imagine a class of students all starting a puzzle by making the exact same guess. They would tend to repeat the same mistakes and learn little from one another. Random initialization gives each “connection” in a neural network a tiny, slightly different starting value instead.
Before the network has seen any examples, it has no useful knowledge. These random starting points are not intelligence; they are simply a fair starting arrangement. Because different parts begin a little differently, they can notice and learn different patterns—such as edges in photos, word meanings, or clues that help spot spam. Good random initialization helps learning begin smoothly rather than getting stuck or becoming unstable.