Batch Size

Think of training as teaching a network from a stack of examples. Batch size is how many examples it examines together before calculating a gradient and updating its weights. Rather than changing the model after every single example—or waiting to inspect the entire dataset—the training loop works in these manageable groups.

What happens in one batch

For a batch of 32 examples, the network runs a forward pass for all 32, computes each example’s loss, combines them (usually by taking their mean), and backpropagates one gradient through the model. The optimiser, such as Adam or SGD, then makes one parameter update. If a dataset has 32,000 examples, batch size 32 produces 1,000 updates per epoch; batch size 320 produces 100.

Small batches give noisier gradient estimates because each update reflects less data.
Large batches give gradients closer to the full-dataset gradient, making each update more predictable.
The batch must fit in accelerator memory because activations and gradients for every example are retained during backpropagation.

The trade-off during training

Noisy small-batch updates are not purely a defect: their variation can help a model move away from unhelpful regions of the loss landscape. They also provide more updates per epoch. But batches that are too small can make loss curves jump around and training inefficient on GPUs. Large batches use parallel hardware efficiently, but demand more memory and usually require retuning the learning rate. Raising batch size tenfold while leaving all other settings unchanged can make learning crawl; raising the learning rate too aggressively can make loss diverge after a few epochs.

Practical details

Batch size also affects layers such as BatchNorm, which estimates activation statistics from the current batch. Very small batches make those estimates unreliable; architectures such as transformers instead commonly use LayerNorm, which does not depend on other examples in the batch. When memory limits prevent a desired effective batch size, gradient accumulation sums gradients across several small batches and performs one update afterward. This saves memory, though it does not make each forward/backward pass faster.

Batch size is the number of training examples processed together to compute one gradient estimate and perform one parameter update. It sets the trade-off between gradient noise, memory use, and update frequency: smaller batches provide noisier, more frequent updates; larger batches produce smoother gradients but require more memory. It matters because it strongly affects training stability, speed, learning-rate selection, and final generalization.

Imagine a teacher grading a stack of homework. They could read every student's work before giving feedback, or look at a small pile, give feedback, then move to the next pile. Batch size is the number of examples an AI looks at together before it adjusts itself.

For instance, with a batch size of 32, the AI reviews 32 photos and then makes one small improvement to its ability to recognize what is in them. Small batches mean more frequent, slightly noisier feedback. Large batches give a broader view each time but need more computer memory. Choosing a batch size is a practical balance between learning smoothly, learning quickly, and fitting the work on available hardware.