Adagrad

Adagrad is an optimizer that gives each parameter its own pace of learning. Instead of applying one shared learning rate everywhere, it slows down parameters that have received many large gradients and preserves larger updates for parameters that have rarely been touched.

How the update adapts
For parameter θᵢ, Adagrad keeps an accumulator of its past squared gradients:

Gᵢ ← Gᵢ + gᵢ², where gᵢ is the current gradient.
θᵢ ← θᵢ − η · gᵢ / (√Gᵢ + ε), where η is the base learning rate and ε prevents division by zero.

Large or frequent gradients make Gᵢ grow, increasing the denominator and shrinking that parameter’s effective learning rate. A parameter with few nonzero gradients keeps a comparatively large step size. Think of each weight carrying a notebook: every strong correction is recorded, and a fuller notebook makes future corrections more cautious.

Why it is useful—and where it stalls
This behavior is particularly valuable for sparse features. In a model with a large embedding table, a rare token’s embedding receives gradients only when that token appears. Plain SGD can leave it under-trained; Adagrad lets it take meaningful steps when it finally does appear. Frameworks expose this directly, such as torch.optim.Adagrad.

The built-in limitation
Adagrad never forgets: its squared-gradient accumulator only increases. Consequently, every effective learning rate can become extremely small after enough training, causing the loss curve to flatten because the network is no longer moving enough—not necessarily because it has reached a good solution. Raising the base rate delays this decay but can make early training unstable. The accumulator also requires one extra value per parameter, increasing optimizer memory. RMSProp and Adam address Adagrad’s main weakness by using decaying gradient statistics, allowing learning rates to recover as training conditions change.

Adagrad is an adaptive gradient optimizer that scales each parameter’s learning rate inversely with the accumulated sum of its past squared gradients. Parameters with frequent or large gradients receive smaller updates, while infrequently updated parameters retain larger effective learning rates. This helps train models with sparse or unevenly distributed features, but its continually growing accumulator can shrink learning rates until progress stalls.

Imagine several people learning different skills. One is making big mistakes often, so they need lots of practice. Another has already practiced the same thing repeatedly, so tiny adjustments are more useful. Adagrad gives each part of an AI model this kind of personalized pace.

As the model learns from examples, Adagrad lets parts that have received many corrections take smaller steps, while less-tested parts can still make larger changes. This can be especially helpful when some clues are rare, such as an unusual word in a huge collection of text. Its downside is that its steps can eventually become so small that learning slows down too much.