Epsilon-Greedy

Imagine a game-playing agent that has found one move that earns points. Should it keep using that move, or risk trying another that could be much better? Epsilon-greedy is the simplest rule for making that trade-off: usually choose the current best-looking action, but deliberately take a random action a small fraction of the time.

How the rule works

Let ε (epsilon) be a number between 0 and 1. At each decision:

With probability 1 − ε, select the action with the highest current value estimate: exploitation.
With probability ε, select an action at random: exploration.

For example, with ε = 0.1, an agent follows its best-known choice 90% of the time and explores randomly 10% of the time. In a DQN agent, “best” means the action with the largest predicted Q-value, \(Q(s,a)\), for the current state.

Why random detours are valuable

Reward alone does not label the best action in every situation. An agent that never explores can lock onto an early lucky choice and never discover a shortcut. In a grid world, it might repeatedly take a safe path worth +1 while missing a route worth +10. Random exploratory actions generate the missing experience needed to revise value estimates. This is especially important because the agent’s choices determine which states and rewards enter its training data.

Practical behavior and limits

Training commonly starts with a larger ε and gradually reduces it—an epsilon decay schedule. Early exploration maps the environment; later exploitation uses what was learned. Gymnasium examples and many DQN implementations use this pattern.

Its simplicity is also its weakness: epsilon-greedy explores blindly. It can choose obviously bad actions just as readily as uncertain, promising ones. If ε stays too high, performance remains noisy; if it falls too quickly, the agent settles prematurely. Methods such as UCB or Thompson sampling use uncertainty more deliberately, but epsilon-greedy remains a reliable baseline because its mechanism is transparent and easy to control.

Epsilon-greedy is an action-selection strategy that chooses the action with the highest current estimated value with probability 1 − ε, and selects a random action with probability ε. It provides a simple, explicit balance between exploitation of known rewarding actions and exploration of alternatives. Without sufficient exploration, value estimates can remain biased toward actions tried early; excessive exploration reduces reward from the current policy.

Imagine choosing a café. You usually go to your favourite because you know it is good. But every so often, you try somewhere new—you might discover an even better place.

Epsilon-greedy is a simple rule for making that kind of choice. Most of the time, a learning system picks the option that currently seems best. But a small amount of the time—set by a number called epsilon—it deliberately tries a different, sometimes random option.

This matters because always repeating the apparent winner can trap the system with a merely “good enough” choice. Occasional experimentation helps it discover better possibilities while still benefiting from what it has already learned.