Dropout
Dropout is a training-time trick that makes a neural network practice without relying too heavily on any one internal feature. Instead of letting the same collection of neurons always work together, it briefly removes a random subset on each training pass.
What happens inside the network
For each activation in a dropout layer, the network draws a random mask: an activation is kept with probability 1 − p and set to zero with probability p, where p is the dropout rate. With inverted dropout, the kept activations are also scaled by 1 / (1 − p) during training. This preserves their expected magnitude, so inference needs no compensating rescale. In PyTorch, torch.nn.Dropout(p) applies this behavior automatically: it randomly zeros values in training mode and becomes an identity operation in evaluation mode.
Why this regularizes
Each mask creates a slightly different “thinned” network. To keep the loss low across these changing networks, useful representations must be spread across multiple units rather than stored in a fragile, highly coordinated path. This reduces co-adaptation: for example, a classifier cannot depend entirely on one hidden neuron that happens to capture a coincidental pattern in the training set. The effect resembles training many related smaller networks and combining them at inference, though dropout achieves this without storing many models.
Practical trade-offs
Dropout adds noise to gradients, which improves generalization but can slow convergence. A rate around 0.1–0.5 is common; excessive dropout causes underfitting, visible as both training and validation loss remaining high. It must be disabled at inference: leaving a model in training mode makes identical inputs produce inconsistent predictions. Dropout is especially useful in large dense layers, but it is used more carefully with BatchNorm, whose batch statistics already inject noise and can reduce dropout’s benefit. In transformer blocks, dropout commonly appears on attention weights and residual paths, while LayerNorm handles normalization separately.
Dropout is a regularization technique that randomly sets a proportion of neuron activations to zero during each training step. This prevents units from relying on specific companions, forcing the network to learn more distributed, robust representations. At inference, all units are active with appropriately scaled activations. Dropout reduces overfitting and improves generalization, particularly in high-capacity networks trained on limited data.
Imagine a sports team that sometimes has to play without a few of its usual players. To keep winning, the team cannot depend entirely on one star; everyone has to learn to contribute.
Dropout does something similar while an AI system is learning. It temporarily “sits out” random parts of the network during practice. This stops the system from relying too heavily on a few familiar shortcuts in its training examples.
As a result, the network learns more flexible, reliable patterns. It may perform a little less perfectly on its practice material, but it is usually better at handling new photos, messages, or situations it has never seen before.