Notes
🔍

Momentum

Training a supervised model can feel like trying to hike downhill in thick fog: you know the direction of steepest descent (the gradient), but your steps can wobble, overshoot, or get stuck in little dips. Momentum is a simple trick that makes those steps steadier and faster.

What momentum does in gradient descent

In plain gradient descent, each update uses only the current gradient. With momentum, the optimizer keeps a running “velocity” vector: an exponentially decaying average of past gradients. The parameter update combines the new gradient with this stored velocity, so consistent directions build speed while noisy back-and-forth directions cancel out. A common form is:

v = β v + (1-β) g
w = w - α v

Here g is the gradient, α the learning rate, and β (like 0.9) controls how much history you keep.

Why it matters during supervised training

Loss surfaces for models like neural networks or even logistic regression can have narrow valleys and flat regions. Momentum helps by:

  • Reducing zig-zagging across steep directions, so you move smoothly along the valley.
  • Speeding up progress across flat plateaus where raw gradients are tiny.
  • Making mini-batch training less jittery when gradients are noisy.

Ignore momentum and you often need a smaller learning rate to stay stable, which slows training.

Where you’ll see it in practice

In spam detection or churn prediction with neural networks, momentum typically reaches a good solution in fewer epochs than plain SGD. In libraries, it’s exposed directly: PyTorch’s torch.optim.SGD(..., momentum=0.9) or Keras’ SGD(momentum=0.9). Related optimizers like Nesterov momentum and Adam build on the same “velocity” idea.

Momentum is an optimization technique that updates parameters using a running “velocity” (an exponential moving average of past gradients) rather than the current gradient alone, typically: v ← βv + g, θ ← θ − ηv. It accelerates learning in consistent descent directions and damps oscillations in narrow valleys, improving convergence speed and stability when training supervised models, especially with noisy mini-batch gradients.

Think about pushing a heavy shopping cart. If you stop pushing every second to “rethink,” the cart barely moves. But if you keep a steady push, the cart builds speed and rolls more smoothly over small bumps. Momentum in machine learning training is like that steady push.

When an AI model is learning, it makes lots of tiny adjustments to improve its predictions (like a spam filter getting better at spotting junk). Momentum helps those adjustments keep moving in a consistent direction, instead of zigzagging or getting stuck. This often makes learning faster and more stable, especially when the path to a good solution is bumpy.