Monte Carlo Prediction

Imagine judging a route only after reaching the destination: you add up every reward earned along the way, then use that finished trip to improve your expectations for the places you visited. That is the central idea behind Monte Carlo prediction: learn how good states are by averaging what actually happened in complete episodes.

Learning from completed experience

For a fixed policy—the agent’s current rule for choosing actions—Monte Carlo prediction estimates the state-value function, V(s). After an episode ends, the algorithm computes the return from each visited state: the discounted sum of all later rewards. Each state’s value estimate is then moved toward the observed return, commonly by taking the running average of all returns seen from that state.

First-visit MC updates a state only from its first appearance in an episode.
Every-visit MC updates it after every appearance; repeated visits each supply a sample.

Why the full episode matters

Unlike temporal-difference methods such as TD(0), Monte Carlo prediction does not update one estimate using another estimate. It waits for the real outcome, so its targets do not “bootstrap.” This makes the basic method conceptually clean: if episodes finish, returns are bounded, and the policy keeps generating representative experience, sample averages converge to the policy’s true values. The trade-off is delay. A robot episode that lasts twenty minutes cannot improve its earliest value estimate until the run ends, and tasks with no natural terminal state need an artificial cutoff or a different method.

What it reveals in practice

In the Blackjack example from Sutton and Barto, an agent plays many complete hands, records the final win or loss, and averages returns for states such as “player total 20, dealer shows 6.” In a Gymnasium environment, the same approach can evaluate a policy after full game rollouts. It also exposes reward-design mistakes: if an agent discovers a shortcut that earns reward while avoiding the intended goal, completed returns faithfully raise the shortcut’s value. Monte Carlo prediction evaluates the rewards supplied; it cannot infer the designer’s unstated intention.

Monte Carlo prediction estimates a policy’s value function by averaging the actual returns observed after states are visited in complete episodes. It requires no model of environment dynamics and can use first-visit or every-visit updates. It matters because it learns directly from reward experience, providing value estimates for policy evaluation, but updates only after episodes terminate.

Imagine learning which route home is best by trying different routes all the way to the end. After each trip, you ask: “How good was that journey overall?” A quick, easy trip gets a high score; a slow, frustrating one gets a low score. Over many complete trips, you build a better sense of which choices tend to lead to good outcomes.

Monte Carlo Prediction does this for an AI. It estimates how promising a situation is by looking at the rewards received after reaching the end of many attempts. It does not need a map of what will happen next. It learns from finished experiences, much like judging a movie only after you have watched it all.