Notes
🔍

Layer Normalization

Layer Normalization gives each example’s internal activations a steadier scale as they pass through a network. Think of it as re-centering and re-scaling the values inside one layer so that a few unusually large numbers do not make the next computation hard to control.

What it normalizes
For one input example, Layer Normalization calculates the mean and variance across that example’s feature dimensions, then transforms each activation:

  • Subtract the layer’s mean, making the activations centered around zero.
  • Divide by the layer’s standard deviation, giving them a controlled spread.
  • Apply learned scale (γ) and shift (β) parameters, so the network can restore any useful magnitude or offset.

Unlike Batch Normalization, it does not combine statistics from different examples in a minibatch. Each example is normalized independently, so its behavior is the same during training and inference.

Why training becomes steadier
Deep layers continually receive inputs whose scale changes as earlier weights are updated. Layer Normalization reduces this shifting target, keeping activations and gradients in a more manageable numerical range. In a transformer block, LayerNorm is placed around the attention and feed-forward sublayers; a common “pre-norm” design applies it before each sublayer. Together with a ResNet-style skip connection, this gives gradients a reliable route through many blocks rather than allowing them to fade or explode.

Practical use and limits
In PyTorch, torch.nn.LayerNorm normalizes the trailing dimensions specified by normalized_shape; for a token representation of width 768, that is typically LayerNorm(768). It adds a small amount of computation and stores per-example statistics, but avoids batch-dependent behavior that can be troublesome with tiny or variable batch sizes. It does not replace a sensible learning rate: a loss curve that suddenly diverges can still signal overly aggressive optimizer updates. LayerNorm stabilizes the terrain; Adam or another optimizer still determines how boldly the model moves across it.

Layer Normalization normalizes a neural network layer’s activations for each individual example, using the mean and variance computed across its feature dimensions, then applies learned scale and shift parameters. Unlike batch normalization, it does not depend on batch statistics. It stabilizes activation scales and gradient flow, enabling reliable training with small or variable batch sizes and in sequence-based architectures.

Imagine a singer adjusting their own microphone before each song, so their voice is neither too quiet nor painfully loud. Layer Normalization does something similar inside an AI network.

As the network processes one piece of information—such as a sentence or image—it keeps the signals within a layer on a steady, manageable scale. This prevents some signals from overwhelming the others and helps the network stay reliable while learning. It is especially useful for AI that handles sequences, such as language models, because each sentence can be treated consistently on its own.