Decaying Epsilon

Early in training, an agent is ignorant: its “best” action is only the best among a handful it happened to try. Decaying epsilon gives it permission to be curious at first, then gradually makes it more decisive as its experience becomes useful.

How the schedule works

It is used with an epsilon-greedy policy. At each decision:

With probability ε, the agent chooses a random action, exploring.
With probability 1 − ε, it chooses the action with the highest current value estimate, exploiting.

Instead of keeping ε fixed, decaying epsilon lowers it over training. A common linear schedule moves from, say, 1.0 to 0.1 over a chosen number of environment steps. An exponential schedule reduces it more quickly at first. Many implementations stop at a nonzero epsilon floor, such as 0.01, so the agent retains some exploration.

Why begin random and become selective

Consider a DQN agent in a Gymnasium game. At the start, its Q-values are poorly trained neural-network outputs, so greedily following them is little better than following noise. High ε produces varied transitions: missed jumps, successful jumps, unusual states, and rewards that reveal which actions matter. Later, once the agent has evidence that one action reliably pays off, continuing to act randomly would throw away reward and slow learning. DQN implementations commonly anneal ε while filling and learning from a replay buffer.

What can go wrong

The decay rate is a practical design choice, not a cosmetic setting. If ε drops too fast, the agent can lock onto a mediocre route because it never discovers a better one—such as repeatedly taking a safe long path while missing a rewarding shortcut. If it stays high too long, its data remain noisy and its achieved reward stays low. In environments whose dynamics shift over time, decaying ε all the way to zero is especially risky: the agent loses the random trials needed to notice that its previously best action has stopped working. Decaying epsilon therefore manages the central reinforcement-learning trade-off between collecting information and using what has already been learned.

Decaying epsilon is an exploration schedule in epsilon-greedy reinforcement learning that gradually reduces the probability, ε, of choosing a random action as training progresses. Early high ε promotes broad exploration; later low ε favors exploiting actions with the highest estimated value. It matters because it shifts learning from information gathering toward reward maximization while retaining limited exploration to avoid premature convergence on poor policies.

Imagine learning which café makes the best coffee. At first, you try lots of different places, even when you already have a favourite. Later, once you have a good idea, you mostly return to the café that has worked out best.

Decaying epsilon is this same idea for a learning program. “Epsilon” is simply the chance that it will try something new instead of choosing what currently seems best. Early on, that chance is high, so it explores. As it gains experience, epsilon gradually shrinks—or decays—so it relies more on its proven choices while still occasionally discovering a better one.