Notes
🔍

Policy Evaluation

Imagine committing to a particular way of behaving—such as “always take the shortest visible route”—and asking: how good is this behaviour from every possible situation? Policy evaluation answers that question before trying to improve the behaviour itself.

What is being estimated
For a fixed policy π, policy evaluation computes its state-value function, written Vπ(s). This is the expected total discounted reward the agent will receive when it starts in state s, follows π from then on, and the environment evolves according to its known dynamics. In dynamic programming, those dynamics are supplied as transition probabilities and expected rewards—not learned by trial and error.

How repeated backups work
The value of a state depends on the immediate reward and on the values of states likely to follow it. Policy evaluation repeatedly applies the Bellman expectation equation:

V(s) ← Σₐ π(a|s) Σₛ′,r p(s′, r | s, a) [r + γV(s′)]

Starting from arbitrary value estimates, each update “backs up” information from successor states. With a finite state space, a fixed policy, and discount factor γ < 1, repeated full sweeps converge to the policy’s true values. A grid-world agent, for example, can estimate how much return its current navigation rule produces from each square, accounting for walls, slippery movement, and terminal rewards.

Why it matters
Policy evaluation is the assessment half of policy iteration:

  • Evaluate: estimate how well the current policy performs.
  • Improve: choose actions that look better using those estimates.
  • Repeat until improvement changes nothing, yielding an optimal policy.

Without accurate evaluation, an improvement step can prefer an action because its downstream consequences were mispriced. Full evaluation can be expensive, so modified policy iteration performs only a few update sweeps before improving. This same evaluate-and-improve loop underlies more practical methods: a critic in actor-critic algorithms such as PPO plays the evaluation role, though it learns from sampled experience rather than a complete known model.

Policy evaluation computes the value function for a fixed policy: the expected discounted return from each state when that policy is followed. In dynamic programming, it repeatedly applies the Bellman expectation equation using known transition and reward dynamics until values converge. These estimates matter because policy improvement depends on accurately comparing the long-term consequences of actions under the current policy.

Imagine you have a fixed set of rules for choosing a route to work: always take the same streets in the same situations. Policy evaluation is like asking, “How good is this route plan likely to be over time?”

It does not try to invent a better plan yet. Instead, it estimates what you can expect if you keep following the current one: which situations lead to smooth trips, which lead to traffic, and how worthwhile each choice is in the long run.

For an AI learning through rewards, policy evaluation is the “how well are we doing?” step. That assessment helps it later decide whether to keep its current behaviour or improve it.