Softmax (Boltzmann) Exploration

When an agent has several actions to choose from, it needs a way to try promising alternatives without blindly ignoring what it has learned. Softmax (Boltzmann) exploration does this by making higher-valued actions more likely, rather than declaring a single action best or choosing randomly with equal probability.

How the probabilities are formed
Given estimated action values, such as a DQN’s Q-values, softmax assigns action a a probability:

P(a) = exp(Q(a) / τ) / Σ_b exp(Q(b) / τ)

Here, τ is the temperature. A high temperature flattens the probabilities, so the agent explores broadly. A low temperature concentrates probability on the action with the largest estimated value, approaching greedy choice. Unlike epsilon-greedy, which gives every non-greedy action roughly the same chance, softmax distinguishes between “nearly as good” and “currently looks much worse.”

What it looks like in practice
Suppose an agent estimates that three actions are worth 10, 9.8, and 2. With a suitable temperature, it will frequently test the second action because it is plausibly competitive, while rarely wasting steps on the third. This is useful when early value estimates are uncertain and a small apparent advantage should not become an absolute commitment. Training can begin with a higher temperature and gradually lower it, shifting from information gathering toward reward collection.

Why its details matter
Softmax exploration belongs directly in the action-selection part of the reward-learning loop: its choices determine which transitions and rewards the agent gets to learn from. Poor temperature settings create familiar failures:

• Too low: an early lucky estimate dominates, and the agent never discovers a better action or a rewarding shortcut.
• Too high: the agent keeps taking weak actions and fails to capitalize on what it has learned.
• Unscaled values: because exponentials amplify differences, changing reward scale can make the same temperature either nearly random or nearly greedy.

For this reason, implementations usually stabilize the exponentials numerically and treat temperature as a carefully tuned or scheduled parameter.