Policy Improvement
Imagine you already have a reasonable plan for navigating a maze: from each location, you know how good that plan is expected to be. Policy improvement is the step where you use that knowledge to revise the plan—choosing actions that look better than the ones the current plan prescribed.
How the improvement step works
For a current policy π, dynamic programming first evaluates it, producing a value Vπ(s): the expected total future reward from state s when following π. Policy improvement then considers every available action a in that state and calculates its one-step look-ahead value:
qπ(s, a) = Σs',r p(s', r | s, a) [r + γVπ(s')]It replaces the old choice with an action that maximizes this quantity. In plain terms: take an action, account for its immediate reward, then add the value of where it is expected to lead. This requires the environment’s transition and reward model to be known.
Why changing the policy is safe
The policy improvement theorem gives this step its power. If the new policy chooses actions whose look-ahead values are at least as good as the current policy’s choices in every state, the new policy is guaranteed to be no worse—and is better wherever it makes a genuinely better choice. A gridworld agent, for example, might discover that moving right toward a goal yields a slightly lower immediate reward than moving up, but reaches states with much greater long-term value. Improvement changes its route accordingly.
Its role in planning
Policy iteration alternates between two pieces: evaluate the current policy, then improve it greedily using those evaluations. Repeating this process reaches an optimal policy in a finite tabular environment. The same idea also underlies generalized policy iteration, where evaluation and improvement need not finish completely before influencing each other. Without improvement, values merely score a fixed behavior; they never turn that score into better decisions.
Policy improvement updates a policy by choosing, in each state, an action with the highest expected return according to the current value function—typically the greedy action under \(q_\pi(s,a)\). The resulting policy is guaranteed to be at least as good as the previous one when values are exact. It matters because alternating policy evaluation and improvement drives policy iteration toward an optimal policy.
Imagine revising your usual route to work. After noticing that one street is consistently faster than your normal choice, you start taking it instead. You have improved your plan using what you learned about the possible routes.
Policy improvement is the same idea for a decision-making system. A policy is simply its usual rule for what to do in each situation. The system looks at how well its current choices are expected to turn out, then updates its rule to choose better options where it can. Repeating this process turns an okay plan into a stronger one, step by step.