Learning Rate Scheduling

Training a model with gradient descent is a bit like walking downhill in fog: if your steps are too big you bounce around and miss the valley, and if they’re too small you creep along and waste time. Learning rate scheduling is the idea of changing that step size as training progresses, instead of keeping it fixed.

What it is and how it works

The learning rate controls how much model parameters move in response to each gradient update. A learning rate schedule defines a rule for adjusting it over training steps or epochs. The usual pattern is: start larger to make fast progress early, then reduce the learning rate to “settle in” and fine-tune near a good minimum. Common schedules include:

Step decay: drop the learning rate by a factor every N epochs.
Exponential decay: multiply by a constant < 1 each epoch/step.
Cosine annealing: smoothly decreases following a cosine curve.
Reduce on plateau: lower it when validation loss stops improving.
Warmup: begin with a small rate, ramp up briefly, then decay.

Why it matters in supervised learning

Schedules directly affect convergence and generalization. Too high for too long can keep loss noisy (e.g., a churn classifier never stabilizes). Too low too early can trap training in a mediocre solution (e.g., a house-price regressor underfits even after many epochs). A good schedule often reaches better validation performance with fewer epochs than a fixed learning rate.

What you’ll see in practice

In PyTorch, schedulers like torch.optim.lr_scheduler.ReduceLROnPlateau or CosineAnnealingLR are common; in Keras, tf.keras.optimizers.schedules and callbacks like ReduceLROnPlateau are widely used. These tools make the learning rate a first-class part of the training loop, not a one-time guess.

Learning rate scheduling is the practice of changing the optimizer’s learning rate over training according to a predefined rule or feedback signal (e.g., step decay, cosine annealing, warmup, or reduce-on-plateau). It matters because the learning rate strongly controls convergence speed and stability: good schedules reach lower loss and better generalization, while poor schedules cause slow training, oscillation, or divergence.

Imagine you’re hiking down a foggy mountain toward a campsite. At first you take big steps to make quick progress, but as you get close you switch to smaller, careful steps so you don’t overshoot and wander past it. Learning rate scheduling is that idea for training AI models.

When a model learns from examples, it makes many small updates to improve. The “learning rate” is the step size of those updates. A schedule changes that step size over time—often starting larger for speed, then shrinking for precision—so training is faster, steadier, and more likely to land on a good solution.