Orthogonal Initialization
Think of a deep network at the instant before it has learned anything: every layer is passing a signal onward and receiving a correction backward. Orthogonal initialization starts the weight matrices in a form that preserves the size and independence of those signals, giving training a stable opening rather than a chaotic one.
What “orthogonal” means
A square matrix W is orthogonal when WTW = I: its columns are perpendicular unit vectors. Multiplying a vector by such a matrix rotates or reflects it without changing its length. For a rectangular neural-network weight matrix, the initializer makes as many rows or columns orthonormal as its shape permits. Its key property is that the matrix’s singular values begin near 1. Therefore, a signal’s norm—and the norm of a gradient passing through the transposed matrix—does not grow or shrink purely because of that linear transformation.
Why this helps training
Across many layers, small scaling errors compound: repeated shrinking produces vanishing gradients, while repeated expansion produces exploding activations or gradients. Orthogonal weights directly address this linear part of the problem. They are especially valuable in deep linear networks and recurrent networks, where repeatedly applying the same transition matrix can otherwise rapidly erase or amplify information. A nonlinear activation still changes signal statistics, so an orthogonal matrix is usually paired with a suitable gain: for example, PyTorch’s torch.nn.init.orthogonal_ accepts a gain chosen for the activation. ReLU networks commonly use a gain near √2.
Practical limits and use
Orthogonal initialization is a starting condition, not a guarantee of stable training. Biases, nonlinearities, normalization, residual paths, optimizer settings, and data all reshape the dynamics after the first update. In a ResNet or transformer, residual connections and LayerNorm provide additional stability; in an RNN, an orthogonal recurrent matrix can preserve information across more time steps. It costs a little more than drawing independent random values because it typically uses a matrix decomposition, but this cost is paid once at initialization. When a deep model’s loss diverges immediately or early-layer gradients are near zero, initialization scale and gain are among the first settings worth checking.
Orthogonal initialization initializes a weight matrix so its rows or columns are orthogonal, preserving vector norms as signals pass through a layer. For square matrices, this means the weights satisfy WᵀW = I. It helps keep activations and gradients well scaled across depth, reducing vanishing or exploding signals and improving early training stability, particularly in deep and recurrent networks.
Imagine setting up a row of mirrors so each one reflects light in a different direction, without crowding or dimming the others. Orthogonal initialization gives a new neural network a similarly well-balanced starting arrangement.
Before learning begins, the network’s adjustable settings need initial values. With orthogonal initialization, those values are chosen to be as independent as possible from one another. This helps information travel through many layers without becoming faint, distorted, or overwhelmingly large.
It is especially useful in deeper networks, where a poor starting setup can make learning slow or unstable. It does not teach the network the answer; it simply gives learning a steadier launchpad.