REINFORCE

Imagine learning to play a game without being told which move was right—only whether the final outcome was good or bad. REINFORCE is a direct way to turn those outcomes into learning signals: actions that preceded high reward become more likely, while actions that preceded poor reward become less likely.

How the update works

REINFORCE is a Monte Carlo policy-gradient algorithm. A policy network, written as π_θ(a|s), chooses actions from states. The agent runs a complete episode, calculates the return G_t—the discounted sum of rewards after each action—and adjusts the parameters θ using:

θ ← θ + α · G_t · ∇θ log πθ(a_t | s_t)

The gradient term asks, “How can this network make the chosen action more probable?” Multiplying it by G_t supplies the verdict. High return reinforces that sampled decision; low or negative return discourages it. This works even when actions are continuous, where choosing the largest value from a list—as in DQN—is not practical.

Credit assignment and variance

Its central weakness is high variance. Every action in a successful episode receives credit, including irrelevant or lucky ones. A robot might reach a target after an accidental shove, and REINFORCE could reinforce the shove along with the useful movements. A standard repair subtracts a baseline, commonly a learned value estimate V(s):

θ ← θ + α · (G_t − V(s_t)) · ∇θ log πθ(a_t | s_t)

The quantity G_t − V(s_t) is an advantage estimate: it rewards actions that did better than expected, without changing the expected gradient. This idea leads directly to actor–critic methods.

Why it matters in practice

REINFORCE is conceptually important because it shows how reward alone can train a policy without labelled “correct actions.” In practice, its episode-by-episode updates are data-hungry and noisy, especially with sparse rewards or long tasks. Modern methods such as PPO retain its policy-gradient foundation but use value baselines, batches of trajectories, and constrained updates to avoid a policy improving briefly and then collapsing. REINFORCE remains a clean starting point for small Gymnasium tasks and for understanding why policy gradients need careful variance control.

REINFORCE is the original Monte Carlo policy-gradient algorithm. It updates a parameterized policy by increasing the probability of actions that produced high observed returns and decreasing it for low-return actions, using complete sampled episodes without requiring a model of the environment. It matters because it provides a direct way to optimize behavior from reward alone, though its high-variance gradient estimates motivate baselines and advantage methods.

Imagine learning a new video game without a guide. You try different moves, and after each round you see whether they helped you score points or made you lose. Over time, you become more likely to repeat moves that led to good results.

REINFORCE is an AI learning method built around that idea. It lets an AI try actions, look at the reward it eventually received, and strengthen its preference for the choices that seemed to help. Choices followed by poor results become less appealing. This is useful when nobody can provide the “correct” move in advance—only the outcome can show whether a decision was worthwhile.