Notes
🔍

Batch Normalization

Batch Normalization, usually called BatchNorm, helps a deep network keep its internal signals in a workable range while its weights are changing. It is like recalibrating an instrument as conditions shift: the network still learns its own useful signals, but avoids being thrown off by wildly changing scales.

What the layer does
During training, BatchNorm receives activations from a mini-batch and calculates their mean and variance. It standardizes each feature by subtracting the batch mean and dividing by the batch standard deviation (with a small epsilon for numerical safety). It then restores flexibility with two learned parameters: gamma, which scales the result, and beta, which shifts it. So BatchNorm does not force every feature to remain zero-mean and unit-variance; it gives the model a stable starting point from which it can learn an appropriate scale and offset.

Training versus inference
At inference time, a single example may arrive alone, so there is no representative batch from which to calculate statistics. BatchNorm therefore uses running mean and variance accumulated during training instead. In PyTorch, nn.BatchNorm1d and nn.BatchNorm2d implement this behavior. For convolutional activations, the statistics for each channel are computed over the batch and spatial positions.

Why it changes training
BatchNorm smooths optimization enough to support larger learning rates and reduces sensitivity to weight initialization. Its batch-dependent noise also provides a modest regularizing effect, reducing the need for as much dropout in some architectures. A common pattern is convolution or linear layer, then BatchNorm, then activation. But it has real constraints:

  • Very small or inconsistent batches produce noisy statistics, making training and inference disagree.
  • Using training mode during evaluation makes predictions depend on the other examples in the batch.
  • In residual networks such as ResNet, BatchNorm is a major reason very deep stacks train reliably.

Batch Normalization normalizes a layer’s activations using the mean and variance computed across a mini-batch, then applies learned scale and shift parameters. During inference, it uses running estimates of these statistics. It stabilizes activation distributions, improves gradient flow, permits higher learning rates, and accelerates training; the batch-dependent statistics also introduce a modest regularizing effect.

Imagine a teacher checking a class’s work and noticing that one day everyone’s answers are written tiny, while the next day they are huge and messy. Before judging the answers, the teacher makes the scale more consistent. Batch Normalization does something similar inside an AI network.

As the network learns, the signals passing between its layers can become uneven or wildly different in size. Batch Normalization keeps those signals in a steadier, more manageable range by looking at a small group of training examples at once. This helps the network learn faster and more reliably, rather than constantly having to adapt to shifting conditions.