Adadelta

Adadelta is an optimizer designed to solve a practical frustration: a single fixed learning rate rarely suits every parameter throughout training. Rather than requiring you to choose a global step size, it uses each parameter’s recent update history to determine an appropriately scaled next step.

How its updates are scaled
Like RMSProp, Adadelta keeps an exponentially decaying average of squared gradients, written informally as E[g²]. A parameter receiving consistently large gradients gets a smaller effective update; one with smaller gradients can move farther. But Adadelta adds a second running average: the squared parameter updates themselves, E[Δθ²]. Its update is roughly:

• Compute the gradient g from backpropagation.
• Update the running average of g².
• Scale the gradient by the ratio of the root-mean-square of past updates to the root-mean-square of recent gradients.
• Record the new update in E[Δθ²].

Why this was useful
Adagrad also adapts each parameter’s scale, but its accumulated squared gradients only grow. Its effective learning rates can therefore shrink until a network barely learns. Adadelta “forgets” old gradients through a decay factor, usually called rho, avoiding that permanent slowdown. Tracking update magnitudes also gives the update the same units as the parameter, a useful dimensional correction. In frameworks such as PyTorch, torch.optim.Adadelta exposes rho, eps, and a learning-rate multiplier; its default learning rate is commonly 1.0 because the adaptive ratio performs most of the scaling.

Training behavior and limits
Adadelta can stabilize a run where raw SGD oscillates because different layers produce gradients on very different scales. Its two state buffers cost roughly two extra tensors per trainable parameter, similar to Adam. It is less common today because AdamW usually reaches strong results with more predictable tuning. If rho is too high, stale statistics make adaptation sluggish; too low, the scaling becomes noisy. A loss curve that plateaus can reflect updates becoming too small, while sharp divergence still signals an excessively large multiplier or unstable gradients.