Learning Rate Warmup

At the very start of training, a network is like a machine whose moving parts have not yet settled into rhythm. Learning rate warmup gives it a gentle start: instead of immediately taking full-sized parameter updates, training begins with small updates and deliberately ramps them up.

How the ramp works

Warmup is a short early phase of a learning-rate schedule. For a chosen number of optimizer steps, the learning rate rises from a small value—sometimes zero—to the intended peak learning rate. A linear warmup, for example, uses:

learning_rate = peak_learning_rate × current_step / warmup_steps

After that point, the main schedule takes over, such as cosine decay or a constant rate followed by decay. In transformer training, a common pattern is linear warmup followed by inverse-square-root or cosine decay.

Why early updates need restraint

Initial gradients can be unusually erratic because weights, activations, and optimizer statistics are still finding sensible scales. This is especially important with Adam: its running estimates of gradient magnitude have very little history during the first few steps. A full learning rate can therefore produce a disproportionately large update, pushing activations into unstable ranges and causing loss to spike or diverge.

Warmup stabilizes the first updates while optimizer momentum and variance estimates become reliable.
Warmup enables larger peak rates, which can make later training faster.
It is particularly valuable in large-batch training and deep residual or transformer-style networks with LayerNorm.

Reading it in practice

If loss explodes in the first few hundred steps but training is stable when resumed at a lower rate, insufficient warmup is a likely cause. Increase warmup steps, lower the peak rate, or both. Excessive warmup does not usually crash a run, but wastes early training time because updates remain too small. A PyTorch training loop can implement this with LinearLR followed by another scheduler; many transformer training utilities provide a combined warmup-and-decay scheduler directly. Warmup does not fix a fundamentally excessive learning rate—it makes the transition to a sensible one safe.

Learning rate warmup is a schedule that starts training with a small learning rate and increases it gradually to a target value over an initial number of steps or epochs. It prevents unstable, oversized parameter updates while gradients, optimizer state, and activation statistics are still settling. Warmup is especially important for deep networks and adaptive optimizers, enabling stable early training before the main learning-rate schedule begins.

Imagine learning to ride a bicycle. You would not begin by racing downhill at full speed. You would start slowly, get your balance, then move faster once you feel steady.

Learning rate warmup gives an AI system that same gentle start. The learning rate is how big a change the system makes after each practice example. During warmup, it begins with very small changes and gradually increases them over an early period of training.

This helps prevent wild, unstable early adjustments, especially in large or complex AI systems. Once the system has found its footing, it can learn more quickly and reliably.