Notes
🔍

Group Normalization

Group Normalization is a way to keep a network’s internal signals on a sensible scale without depending on how many examples fit in a training batch. It is particularly useful when memory limits force very small batches, where batch-based statistics become noisy and unreliable.

How it normalizes activations
For each individual example, Group Normalization (GN) splits a layer’s channels into a chosen number of groups. Within each group, it calculates the mean and variance across that group’s channels and spatial positions, then rescales the values to have a stable distribution. Like other normalization layers, it also learns a scale and offset, so the network can retain useful signal ranges when needed.

Why groups are useful
GN sits between several familiar choices:

  • Batch Normalization computes statistics across examples in the batch. It works well with large, consistent batches but becomes unstable with tiny batches.
  • Layer Normalization uses all channels of one example together.
  • Instance Normalization effectively gives every channel its own statistics.
  • Group Normalization normalizes small channel collections, preserving a middle ground between treating all channels alike and treating each one independently.

Training behavior and practical use
Unlike BatchNorm, GN produces the same kind of normalization during training and inference because it never relies on batch-level running averages. This makes behavior predictable for small-batch convolutional networks, such as detection or segmentation models, and for distributed training where each device sees few examples. In PyTorch, it appears as torch.nn.GroupNorm(num_groups, num_channels); a common configuration uses 32 groups when the channel count permits it. Too few groups makes GN resemble LayerNorm; one channel per group makes it resemble InstanceNorm. GN adds a small amount of computation for per-group statistics, but avoids the training instability and inaccurate inference statistics that BatchNorm can introduce when batches are small.

Group Normalization normalizes a layer’s activations independently for each example by dividing channels into groups and computing a mean and variance within each group across its spatial dimensions. Unlike Batch Normalization, it does not depend on batch-wide statistics. It stabilizes activation scales and gradients when batch sizes are small, variable, or unsuitable for reliable batch statistics.

Imagine a class where students are practising in small study groups. Instead of comparing every student with the whole class, each group checks whether its own work is roughly on track. Group Normalization does something similar inside an AI image-processing network.

It divides a layer’s many visual “channels” — separate detectors for things like edges, colours, or textures — into small groups. Each group is kept on a similar, manageable scale. This helps the network learn steadily rather than becoming thrown off by wildly different signal sizes.

It is especially useful when the AI must train with only a few examples at a time, such as when processing large, detailed images.