Zero Initialization

Setting every parameter to zero looks like the safest possible starting point: no layer begins with a strong preference. In a deep neural network, though, that apparent neutrality creates a serious problem. Neurons that start identically receive identical signals, produce identical outputs, and get identical updates—so they remain copies of one another instead of learning different features.

Why learning gets stuck in symmetry
Consider a dense layer with several neurons, each computing activation(weighted input + bias). If every weight and bias is zero, every neuron sees the same input and produces the same activation. During backpropagation, the loss supplies the same gradient to each neuron, so an optimiser such as Adam or SGD changes their weights by exactly the same amount. A layer with 256 nominally separate neurons then behaves like one neuron duplicated 256 times. Depth does not fix this: the symmetry propagates through the network.

What gradients reveal
For a weight connecting input feature x to a neuron, its gradient is roughly the incoming error signal multiplied by x. Zero weights do not necessarily make that particular gradient zero, but they make gradients indistinguishable across neurons. The network can move away from zero, yet it cannot break the “all neurons are the same” pattern by itself. With certain architectures or activations, zero values also block gradient flow entirely. For example, a ReLU unit at zero has no useful negative-side gradient, making an all-zero hidden layer especially risky.

Where zeros are useful

Biases are commonly initialized to zero because random weights already distinguish the neurons.
A final prediction layer can be initialized to zero in carefully designed setups; it initially outputs a neutral prediction, though upstream learning can be delayed for one update.
Some ResNet designs zero-initialize the final scale in a residual branch, so each block begins close to an identity connection. This stabilizes very deep training without making all parallel neurons identical.

Random initializations such as Xavier/Glorot and He/Kaiming solve the core issue by assigning small, different values while also keeping activations and gradients at workable scales. Zero is therefore a valuable surgical tool, not a general-purpose weight initialization.

Zero initialization sets all or selected trainable parameters to zero before training. It is appropriate for biases and certain residual or normalization parameters, but initializing all weights in a layer to zero preserves symmetry: neurons produce identical outputs and receive identical gradients, so they remain identical and cannot learn distinct features. Breaking this symmetry with varied weight initialization is essential for effective representation learning.

Imagine asking a team of students to solve a puzzle, but giving every student the exact same starting clue. They will all make the same guesses and never discover different parts of the answer.

Zero initialization means starting a network’s adjustable settings, called weights, at zero. This sounds tidy, but for most network layers it is a bad starting point: the units begin identical, learn identical things, and fail to divide up the job. The network loses the benefit of having many different “eyes” looking for different patterns. Zeros can still be useful for some settings, such as small adjustment values called biases, but weights usually need varied starting values.