Gradient Accumulation

Large batches can make training smoother, but they also demand more GPU memory than a model can fit. Gradient accumulation provides a practical workaround: the model processes several small batches, keeps adding their learning signals together, and updates its weights only after the group is complete.

How the update is built
A small batch processed in one forward and backward pass is a microbatch. Rather than calling the optimiser after every microbatch, training performs these steps:

Run a forward pass and compute the loss for one microbatch.
Scale that loss by the number of accumulation steps, so the final gradient represents an average rather than an oversized sum.
Run backpropagation; its gradients are added to the gradients already stored on each parameter.
After, for example, 8 microbatches, call optimizer.step(), then clear gradients with zero_grad().

A microbatch of 4 with 8 accumulation steps gives an effective batch size of 32. Crucially, the model’s parameters stay fixed during all eight passes; only then do they change.

Why it helps—and what it costs
Accumulation lets a transformer or other memory-hungry network use a larger effective batch without storing activations for all examples simultaneously. It is especially useful when a training run is noisy or unstable because the physical batch must be tiny. It does not make computation cheaper: eight microbatches still require eight forward and backward passes, and updates happen less frequently. Learning-rate schedules should count optimiser updates, not microbatches, or a schedule can decay eight times too quickly.

Practical details
In PyTorch, gradients accumulate by default until optimizer.zero_grad() is called. With mixed precision, unscale gradients and apply gradient clipping immediately before the final optimiser step. Gradient accumulation closely reproduces a large batch, but not perfectly when layers maintain batch-dependent state, such as BatchNorm, or when dropout draws different random masks across microbatches. LayerNorm, used in transformer blocks, has no such cross-example batch statistics, making it particularly compatible with this technique.

Gradient accumulation computes gradients over several smaller mini-batches, sums them without updating parameters, then applies one optimizer update using the accumulated gradient. It simulates a larger effective batch size while keeping per-batch memory use low. This enables training models or sequence lengths that would not fit in memory, though the learning-rate and update schedule must be interpreted relative to the effective batch size.

Imagine trying to judge a whole class’s homework, but your desk only has room for a few papers at a time. Instead of making a decision after each small pile, you keep notes from several piles, then make one better-informed decision.

Gradient accumulation does something similar while training an AI model. The model looks at several small groups of examples, collects the lessons from each group, and waits before adjusting itself. This lets it behave as though it had learned from one much larger group, without needing enough computer memory to hold that large group all at once. It helps train bigger models on limited hardware.