Adam

Training a neural network means repeatedly deciding how far to move each weight after seeing its gradient. Adam makes those moves less blunt: it remembers both the direction gradients have been pointing and how erratic they have been, then adjusts each parameter’s step size accordingly.

How Adam turns gradients into updates

For each parameter, Adam maintains two running averages. The first moment (m) is an exponentially decaying average of gradients, similar to momentum: persistent directions build up, while noisy one-off gradients matter less. The second moment (v) is an average of squared gradients, measuring their recent magnitude. Adam updates a weight roughly in the direction of m, divided by the square root of v plus a tiny epsilon. A parameter receiving large or volatile gradients therefore gets a smaller effective step; one with consistently small gradients can move relatively farther.

Why the early steps need correction

Both running averages start at zero, so they are biased toward zero during the first updates. Adam applies bias correction before using them. Its behavior is controlled mainly by:

Learning rate (commonly 0.001 as a starting point): the global scale of every update.
β₁ (commonly 0.9): how much momentum Adam retains.
β₂ (commonly 0.999): how long it remembers squared-gradient scale.

Practical training consequences

Adam is a strong default for deep, noisy optimization problems and appears directly as torch.optim.Adam. It can make a network learn quickly when plain SGD oscillates or when different layers produce gradients on very different scales. It costs extra memory: Adam stores two additional tensors, m and v, for every trainable parameter. A learning rate that is too high still causes loss to spike or diverge; adaptivity is not a safety guarantee. For modern networks, AdamW is frequently preferred because it applies weight decay separately from Adam’s gradient transformation, giving regularization behavior that is easier to control.

Adam (Adaptive Moment Estimation) is a gradient-based optimizer that updates each parameter using exponential moving averages of its gradient (first moment) and squared gradient (second moment), with bias correction. These estimates create adaptive, per-parameter step sizes while retaining momentum. Adam enables stable, efficient training under noisy or unevenly scaled gradients and is a standard default optimizer for deep networks.

Imagine teaching someone to ride a bike. You would not give the same correction every time: if they keep leaning too far left, you focus on that; if their pedaling is already improving, you make smaller changes there. Adam is a popular way for an AI network to make similarly tailored corrections while learning.

As the network studies examples, Adam tracks which parts have been changing consistently and which have been erratic. It then adjusts each part by an appropriate amount: bigger moves where learning seems reliable, gentler moves where things are noisy or unstable. This often helps large networks learn faster and more smoothly than using one fixed adjustment size everywhere.