Lion Optimizer

Training a neural network means repeatedly deciding how to nudge millions or billions of parameters. Lion is an optimiser designed to make those nudges simpler and more memory-efficient than Adam while retaining the useful stabilising effect of momentum.

How Lion chooses an update
Lion’s name comes from Evolved Sign Momentum. Like momentum-based methods, it keeps a running average of recent gradients, which filters out noisy one-batch fluctuations. But instead of using the average gradient’s size to determine each parameter’s step, Lion uses only its sign: positive means step down in one direction; negative means step the other way. In simplified form:

Combine the current gradient with stored momentum.
Take the sign of that combined direction: +1, −1, or occasionally 0.
Move every parameter by the learning rate in that direction.
Update the stored momentum for the next training step.

Why this differs from Adam
Adam stores both a first-moment estimate (a smoothed gradient) and a second-moment estimate (a smoothed squared gradient), using the latter to scale steps per parameter. Lion stores only the momentum-like first estimate, so its optimiser state uses roughly half the memory of Adam. That is valuable when model parameters, gradients, and optimiser state already strain accelerator memory. Its sign-based steps also make the update magnitude controlled directly by the learning rate rather than by changing gradient scales.

Training behavior and practical care
Lion is commonly paired with decoupled weight decay, as in AdamW. It can train large models competitively, but it is not a drop-in guarantee: its learning rate and weight decay need retuning. A rate that worked for Adam can make Lion’s fixed-size directional updates overshoot, visible as a loss curve that oscillates or diverges after a few epochs. Conversely, too small a rate makes progress crawl. In frameworks that expose it as an optimiser, such as a PyTorch Lion implementation, the crucial controls are the learning rate, two momentum coefficients, and weight decay. Lion is best understood as a memory-conscious optimiser that trusts a smoothed gradient’s direction more than its raw magnitude.

Lion (Evolved Sign Momentum) is a gradient-based optimizer that updates parameters using the sign of a momentum-smoothed gradient rather than Adam’s second-moment variance estimate. It maintains a single momentum state and typically uses decoupled weight decay, reducing optimizer memory relative to AdamW. Lion matters because its sign-based updates can train large networks efficiently with lower memory use, though it requires carefully tuned learning rates and weight decay.

Imagine teaching a huge team to improve a recipe. After each taste test, everyone gets a simple instruction: “Use a little more salt,” “Use less sugar,” or “stay as you are.” The Lion Optimizer is a method that gives an AI network similarly direct guidance while it learns.

Training an AI means repeatedly adjusting countless internal settings so its answers become better. Lion looks at the recent direction of improvement and mainly decides which way each setting should move, rather than fussing over the exact size of every correction. This can make learning more memory-efficient, which is especially useful for very large AI models.