Optimal Policy

An optimal policy is the best rule for choosing actions that an agent can have for a particular environment and objective. “Best” does not mean it gets the largest immediate reward; it means its choices produce the greatest expected total reward over the future.

A policy, written as π, maps a state to an action or to probabilities over actions. Its quality is measured by its expected discounted return: the sum of future rewards, with rewards farther away weighted by a discount factor γ. An optimal policy π* achieves at least as much expected return as every other policy from every state:

• V*(s) is the highest achievable expected return starting in state s.
• Q*(s, a) is the highest achievable expected return after taking action a in state s, then behaving optimally.
• Choosing an action with the largest Q*(s, a) produces an optimal policy.

Imagine a navigation agent that can take a small reward now by collecting a nearby token, or travel farther to reach a goal with a much larger reward. A greedy “take the reward in front of me” rule can fail badly. The optimal policy accounts for the fact that an action changes the next state, and therefore changes all later opportunities. In a deterministic setting, it can be written as choosing arg max_a Q*(s, a). In a stochastic environment, multiple actions can tie, and more than one optimal policy can exist.

Algorithms rarely know the optimal policy directly. DQN, for example, learns an approximation to Q* from experience, then acts greedily with respect to that estimate. This is difficult because reward can be delayed, the agent must explore actions whose value it does not yet know, and value estimates are trained from other estimated values. A policy that looks optimal during training can also fail when the environment’s dynamics shift slightly. The term is therefore an ideal target defined by the environment and reward function—not a guarantee that a learned neural policy has truly reached it.