OneCycle Learning Rate

A neural network learns fastest when its step size changes with the stage of training. OneCycle Learning Rate is a schedule that starts cautiously, briefly pushes the learning rate high enough to explore boldly, then steadily shrinks it to let the model settle into a good solution.

How the cycle works
The learning rate follows one broad rise-and-fall pattern across the entire training run, updated after each batch rather than only after an epoch:

• It begins at a small fraction of the chosen maximum learning rate.
• During the first portion of training, it rises to that maximum. These larger updates can move parameters out of unhelpful, narrow regions of the loss landscape.
• It then falls below its starting value, ending at a very small rate. This final “annealing” phase makes fine adjustments instead of continuing to bounce around.

In the original approach, momentum moves in the opposite direction: high while the learning rate is low, and lower near the learning-rate peak. Think of learning rate as how far a runner steps and momentum as how much they keep moving in the same direction: long exploratory steps need less carry-over, while short finishing steps benefit from steadiness.

Why it helps—and what can go wrong
A well-chosen peak rate can train a network quickly and also regularize it: large mid-training updates discourage the model from settling too early into a brittle solution. In PyTorch, torch.optim.lr_scheduler.OneCycleLR implements this schedule, typically alongside SGD with momentum or Adam/AdamW. Its key setting is max_lr; practitioners commonly estimate a safe value with a learning-rate range test. Set it too high and the loss can spike, become NaN, or diverge after the rising phase. Set it too low and training merely crawls. The schedule also needs the correct total number of optimizer steps: ending the cycle early skips the low-rate refinement stage, while extending it leaves the model taking tiny, unproductive updates.