Notes
🔍

Mini-Batch

A mini-batch is a small group of training examples processed together before a neural network updates its weights. Rather than learn from one example at a time or wait to examine the entire dataset, training moves in manageable handfuls: for example, 32, 128, or 512 examples per update.

What happens in one mini-batch
For each mini-batch, the training loop performs the same core cycle:

  • The network makes predictions for every example in the batch at once.
  • The loss function produces a loss per example, usually reduced to a mean batch loss.
  • Backpropagation computes gradients of that mean loss with respect to each parameter.
  • An optimizer such as SGD or Adam uses those gradients for one parameter update.

This is why a batch size is not merely a data-loading setting: it determines how much evidence each update is based on.

Why not use one example or everything?
A batch of one produces a very noisy gradient: one unusual example can point the update in a misleading direction. Using the full dataset gives the exact average gradient, but each update is expensive and requires holding far more activations in memory. Mini-batches strike the useful balance. They average away much of the noise while allowing GPUs to perform many calculations in parallel. That remaining noise is not purely bad; it can help the optimizer avoid settling too quickly into a poor region of the loss landscape.

Practical effects of batch size
A larger batch uses more GPU memory and yields smoother-looking loss curves, but provides fewer updates per epoch. A smaller batch fits more easily in memory and updates more frequently, but makes training measurements jumpier. Changing batch size usually requires reconsidering the learning rate: a rate that is stable at batch size 32 can diverge at 512. In PyTorch, a DataLoader(batch_size=32) groups examples, while the optimizer’s step() is normally called once per group. Batch size also affects BatchNorm: very small batches give it unreliable per-batch statistics, which can destabilize training.

A mini-batch is a small subset of training examples processed together in one forward pass and backward pass to compute a gradient and perform one parameter update. It balances the noisy, inexpensive updates of single-example training against the stable but costly gradients of full-dataset training. Mini-batch size determines memory use, computational efficiency, gradient noise, and the number of updates per training epoch.

Imagine a teacher checking a huge pile of homework. Rather than read every student’s work before giving any feedback, they check a small stack, notice common mistakes, give a quick lesson adjustment, then move to the next stack.

A mini-batch is that small stack of examples used at one time while training an AI. The system looks at a manageable group of photos, sentences, or other examples, sees how well it did, and makes a small adjustment. Repeating this with many different mini-batches lets it learn from a large dataset without needing to handle everything at once. It makes training practical and keeps learning moving steadily.