RMSProp

RMSProp is a way of making gradient descent less jumpy and less blind. Instead of giving every network parameter the same step size, it watches how large that parameter’s recent gradients have been and adjusts its updates accordingly.

How the update adapts

For each parameter, RMSProp keeps an exponentially decaying average of squared gradients:

vₜ = ρvₜ₋₁ + (1 − ρ)gₜ²
θₜ = θₜ₋₁ − η · gₜ / (√vₜ + ε)

Here, gₜ is the current gradient, vₜ is its recent squared magnitude, ρ controls how much history is retained (commonly 0.9), and ε prevents division by zero. A parameter receiving consistently large gradients gets a smaller effective update; one with smaller gradients gets relatively more room to move. This prevents a steep direction from dominating training.

Why it helps training

Plain SGD uses one learning rate everywhere, which can make it bounce across a narrow, steep loss valley while barely progressing along its shallow direction. RMSProp rescales those directions independently. Unlike Adagrad, whose accumulated squared gradients grow forever and can shrink learning rates almost to nothing, RMSProp gradually forgets old gradients. That makes it useful when gradients change during training, such as in recurrent networks or deep models whose later layers evolve as earlier layers learn.

Practical behaviour and trade-offs

RMSProp maintains one extra value per trainable parameter, so it costs roughly another parameter-sized block of optimizer memory, plus small arithmetic overhead. In PyTorch, it is available as torch.optim.RMSprop. If loss suddenly diverges after a few epochs, lowering its base learning rate is the first correction: adaptive scaling does not make an excessively large learning rate safe. If loss plateaus, a learning-rate schedule can restore useful progress. RMSProp can also use momentum, but it is distinct from momentum: its central job is scaling updates by recent gradient magnitude.

RMSProp is an adaptive gradient optimizer that divides each parameter’s update by the root mean square of its recent gradients, using an exponentially decaying moving average. This gives parameters with consistently large gradients smaller effective learning rates and those with smaller gradients relatively larger ones. RMSProp stabilizes training by preventing accumulated gradient magnitudes from shrinking updates indefinitely, improving convergence on non-stationary or unevenly scaled objectives.

Imagine teaching a group of people to adjust a complicated recipe. Some ingredients need tiny changes—a pinch more salt—while others can handle bigger changes, like adding a cup of flour. Using the same adjustment size for every ingredient would be clumsy.

RMSProp is a training helper for AI networks that does something similar. As the network learns from examples, RMSProp notices which internal settings have been changing wildly and which have been changing gently. It encourages careful steps for the unstable ones and allows more confident steps for the steadier ones.

This helps learning stay smoother and faster, especially when the AI is tackling problems whose patterns shift or vary a lot.