AdamW

AdamW is a version of Adam designed to make a familiar form of regularisation behave predictably. It keeps Adam’s useful per-parameter adaptive step sizes, while handling weight decay as a separate “shrink the weights” operation rather than mixing it into the gradient.

What the W changes
Adam tracks two moving averages for each parameter: the average gradient (momentum) and the average squared gradient (its recent scale). Parameters with consistently large or noisy gradients receive differently scaled updates than quiet ones. In ordinary Adam, adding an L2 penalty to the loss does not produce clean weight decay, because Adam rescales that penalty along with every other gradient component. The amount of shrinkage then differs across parameters in an unintended way.

AdamW decouples these jobs. Each training step conceptually does two things:

• Uses Adam’s moment estimates to move parameters in the loss-reducing direction.
• Multiplies the weights by roughly (1 − learning rate × weight decay), gently pulling them toward zero.

This separation makes the weight_decay setting mean what practitioners expect: a direct pressure against unnecessarily large weights. Bias terms and normalisation parameters are commonly excluded from decay, since shrinking them is usually unhelpful.

Why training runs depend on it
AdamW is a standard default for transformer training and appears directly as torch.optim.AdamW. Its decay helps control overfitting and prevents parameter magnitudes from drifting upward, while Adam’s adaptation keeps early training stable across uneven gradient scales. Too much decay forces useful weights toward zero and causes underfitting; too little can hurt validation performance. A large learning rate can still make loss spike or diverge—AdamW is not a cure for unstable updates—so learning-rate warm-up and decay schedules remain important. It costs slightly more memory than SGD because AdamW stores two extra state tensors per trainable parameter.