Identity Initialization
Identity initialization gives a network a calm, sensible starting point: each layer initially passes its input through unchanged rather than immediately scrambling or shrinking it. The basic idea is simple, but it is especially valuable when a model is very deep or repeatedly applies the same transformation over many steps.
How it works
For a square weight matrix W, identity initialization sets it to the identity matrix I: ones on the diagonal and zeros elsewhere. Thus, before learning,
Wx = Ix = x. Each output feature initially copies the corresponding input feature. Biases are usually initialized to zero. This preserves the scale of forward activations and gives gradients a direct route backward through the layer, avoiding the immediate vanishing or explosion caused by repeatedly multiplying by poorly scaled matrices.
Where it is useful
Identity-like starts appear in several practical designs:
- In a recurrent network, an identity recurrent matrix can preserve information across time steps better than a random matrix. The IRNN approach used this idea with ReLU units.
- In a ResNet-style residual block, initializing the residual branch near zero makes the whole block begin close to y = x. The network starts as a stack of safe pass-through paths, then learns useful deviations.
- For layers whose input and output widths differ, exact identity is impossible; a rectangular “partial identity” can copy as many coordinates as fit, while the remaining entries start at zero or a small random value.
Benefits and limits
The main benefit is stable signal propagation at the beginning of training, which can make very deep networks easier to optimize. But identity initialization is not a universal replacement for Xavier or He initialization. A plain stack of identical identity-initialized layers has little initial diversity, and nonlinear activations, normalization, and changing layer widths alter the pass-through behavior. It is most powerful when the architecture itself supports an identity path—especially residual connections—or when preserving state is the explicit goal. In PyTorch, torch.nn.init.eye_ initializes a compatible weight tensor this way.
Identity initialization initializes a layer or residual branch to implement, or closely approximate, the identity mapping: its output initially equals its input. This is commonly achieved by setting residual-branch weights or final scaling parameters to zero, while preserving a skip connection. It makes deep networks begin as stable near-identity transformations, allowing new layers to refine rather than disrupt existing signal flow and improving early gradient propagation.
Imagine setting up a long row of clear windows. At the start, each window lets the view pass through unchanged. Identity initialization gives parts of a neural network a similar starting point: they initially act almost like “do nothing” steps, passing information along without changing it much.
This can be useful in very deep networks, where many layers might otherwise distort or weaken a signal before learning has even begun. Starting near an identity lets the network begin as a stable, simple version of itself. During training, it can then gradually learn which layers need to make useful changes and which should mostly leave the information alone.