Label Smoothing

Label smoothing is a small change to the training targets that makes a classifier less certain that every training label is absolutely unquestionable. Instead of teaching the network “this example is 100% class A and 0% every other class,” it leaves a tiny amount of probability for the alternatives.

How the target changes

For a problem with K classes, ordinary one-hot labels assign 1 to the correct class and 0 to the rest. With smoothing value ε, the target becomes less sharp: the correct class receives 1 − ε, while the remaining probability is distributed across the other classes. For example, with three classes and ε = 0.1, a target of [1, 0, 0] becomes roughly [0.9, 0.05, 0.05]. Cross-entropy loss then rewards high confidence in the correct answer, but penalizes driving its predicted probability all the way to 1.

Why this regularizes training

A powerful network can memorize training examples by producing extreme logits: one class dominates and every alternative is crushed. That behavior fits labels tightly but produces poor probability estimates and brittle decisions. Label smoothing discourages these extreme output distributions, reducing overconfidence and limiting the incentive for the final layer’s weights to grow excessively. It can improve validation accuracy, calibration, and robustness when labels contain ambiguity or occasional mistakes. Unlike dropout, it changes the loss target rather than randomly altering the network; it is active only during training.

Practical use and trade-offs

Modern classifiers commonly use small values such as 0.05 or 0.1. In PyTorch, it is built into torch.nn.CrossEntropyLoss(label_smoothing=0.1). Too much smoothing makes targets vague: the model underfits, accuracy can fall, and it becomes less able to represent genuinely certain predictions. It also changes downstream confidence scores, so a system that needs probabilities for ranking or thresholding should evaluate calibration after training. Label smoothing is particularly useful when a loss curve keeps falling while validation performance stalls: the model may be becoming more confident on training labels rather than more correct.

Label smoothing is a regularization technique that replaces one-hot training targets with softened probability distributions, assigning most probability to the correct class and a small amount to others. It prevents a classifier from becoming excessively confident, reducing overfitting and improving calibration and generalization. Without it, cross-entropy training can drive output logits toward extreme values.

Imagine teaching a child to identify animals. Rather than saying, “This picture is definitely a cat and nothing else,” you might say, “It’s almost certainly a cat, but stay open to small uncertainty.” Label smoothing gives an AI this same gentler lesson.

Normally, training examples come with absolute answers: cat = 100%, dog = 0%. Label smoothing softens those answers slightly. It encourages the network not to become overly certain just because it saw particular training pictures. That matters because real-world images, voices, and text can be ambiguous or different from the examples it practiced on. The result is often an AI that makes more reliable predictions on new data.