Foundations & Concepts

Reinforcement learning studies a different problem from the rest of machine learning. There is no dataset of labelled examples; there is a learner making decisions, a world that responds, and a stream of feedback that says how well things are going without ever saying what the right choice would have been. This section builds the vocabulary and the mathematics that every later method in this chapter assumes.

Core Vocabulary

The setting has two parties. The Agent is the learner and decision-maker — the thing being trained. The Environment is everything else: the world the agent acts in, which responds to what it does. The boundary between them is not physical but informational, drawn wherever the agent's direct control ends.

They interact in a loop. At each Time Step the agent observes the State, a description of how things currently stand, and chooses an Action from those available. The environment responds with a new state and a Reward, a single number scoring what just happened. That number is the entirety of the feedback — it does not say which action would have been better, only how this one turned out, and learning from that thin signal is the whole difficulty of the field.

What the agent is actually learning is a Policy: a rule mapping states to actions, which may be a lookup table, a set of probabilities, or a neural network. Everything else in this chapter exists to improve one.

Two units organise the record of interaction. An Episode is one complete run from a starting state to a terminal one — a single game, a single delivery attempt. A Trajectory is the sequence of states, actions and rewards actually experienced along the way, and it is the raw material every learning algorithm here consumes.

Markov Decision Processes

That informal loop has a precise mathematical form, the Markov Decision Process (MDP), and nearly every method in this chapter is derived against it. An MDP is defined by a handful of pieces. The State Space is the set of situations that can occur and the Action Space the set of choices available; each may be small and countable or continuous and vast, and which it is largely determines the practical methods available.

The Transition Function says how the world moves — given a state and an action, which state follows, possibly with probabilities rather than certainty. The Reward Function says what feedback that produces. Together they are the environment's dynamics, and the crucial thing is that the agent typically does not know either one.

The framework rests on the Markov Property: the current state contains everything relevant about the past, so how the agent arrived there does not matter for what happens next. This is a strong assumption and it is what makes the problem tractable, because it means a policy needs to consider only the present state rather than an ever-growing history.

Two settings complete the picture. The Horizon is how far into the future the problem extends, whether a fixed number of steps or indefinitely. The Discount Factor (Gamma) is a number just below one that shrinks the weight of each successive future reward. It expresses a preference for sooner over later, and it does necessary mathematical work: without it, a total over an unbounded future need not be a finite quantity at all.

Value Functions

Choosing well requires judging not the immediate reward but everything that follows from a choice — a losing move that pays a small bonus now is still a losing move. A Value Function captures that: it estimates the total future reward expected from a situation, under a given policy.

It comes in two forms, and the difference matters constantly. The State-Value Function (V) asks how good a state is if the agent follows its policy from there. The Action-Value Function (Q) asks how good it is to take a particular action in a state and follow the policy afterwards. The second is more immediately useful for control, because comparing its values across the available actions directly tells the agent what to do — no model of the environment required.

These functions obey a recursive relationship, the Bellman Equation, which states that the value of a state equals the immediate reward plus the discounted value of whatever comes next. That single idea — that a long-run quantity can be written in terms of itself one step later — is what makes the problem computable at all, and virtually every algorithm in this chapter is a way of solving or approximating it. The Bellman Optimality Equation is its sharper form, describing the values achieved by acting best rather than by following some particular policy, and its solution defines an Optimal Policy — one whose expected return is at least as good as any other's from every state.

Returns & Rewards

The quantity all of this maximises deserves care. The Return is the total reward accumulated from a point onward — not the reward at a step, but everything that follows it. Its simplest form is the Cumulative Reward, a plain sum, which works for episodes guaranteed to end. The Discounted Return applies the discount factor to each future term, keeping the total finite over an unbounded horizon and expressing that nearer rewards matter more.

Optimising a return raises a genuine difficulty. When a reward arrives long after the decisions that earned it, which of those decisions deserves the credit? This is the Credit Assignment problem, and much of the machinery in later sections exists to address it.

Its severity depends on how feedback is distributed. A Dense Reward arrives frequently, giving near-continuous guidance and making learning comparatively easy. A Sparse Reward arrives rarely — perhaps only once, at the end, as win or lose — which is both the more natural way to state a goal and far harder to learn from, since the agent may act for a long time with no signal whatsoever about whether it is doing well.

RL Paradigms

Reinforcement learning is not one algorithm but a family of them, and the family divides along several independent lines: what the agent learns, what kind of problem it faces, and how many agents are involved at once. These distinctions are worth grasping early, because the labels recur constantly in the sections that follow and they combine — a method is described by where it sits on each axis, not by one label alone.

Learning Approaches

The first division concerns whether the agent tries to understand the world or merely to act well in it. Model-Free RL learns directly from experience without ever building a representation of the environment's dynamics: it does not predict what the next state will be, it simply learns which actions pay. This gives up the ability to plan ahead, and in exchange avoids the hard problem of learning an accurate model — which is why it dominates in practice.

Within that, methods differ in what they learn. Value-Based RL learns a value function and derives behaviour from it, acting greedily with respect to the values it estimates; the policy is a consequence rather than a thing learned directly. Policy-Based RL inverts this and adjusts the policy itself, which handles continuous action spaces naturally and can represent genuinely random behaviour — both awkward for a value-based method that must pick a maximum.

A third axis cuts across those two and causes more confusion than either. On-Policy Learning improves the same policy it uses to gather experience, so its data must come from current behaviour and is discarded once the policy changes. Off-Policy Learning separates the two, learning about one policy while behaving according to another. That separation is what allows an agent to learn from stored past experience, from a replay buffer, or from another agent's behaviour entirely — a large practical advantage, and one that brings its own instabilities.

Problem Settings

The second division describes the problem rather than the method. An Episodic Task has natural endings — a game concludes, a robot completes or fails a grasp — and each episode starts fresh, which gives a clean definition of total reward. A Continuing Task never terminates: a controller regulating a process runs indefinitely, and with no end point the discounting introduced earlier stops being a preference and becomes a necessity.

Environments differ in predictability too. A Deterministic Environment responds to an action the same way every time, so the same trajectory is reproducible. A Stochastic Environment incorporates chance, meaning identical actions may lead to different outcomes and an agent must reason about expectations rather than certainties. Most interesting problems are stochastic, which is why the value functions of the previous section are defined as expected returns.

A more demanding case relaxes the Markov assumption itself. In a Partially Observable MDP (POMDP) the agent cannot see the full state and receives only an observation consistent with several possible underlying situations — a robot with limited sensors, a card game with hidden hands. Since the current observation no longer summarises the past, the agent must accumulate history to act well, which is why memory-bearing architectures appear in this setting.

Multi-Agent RL

The final division adds other decision-makers. Multi-Agent RL (MARL) studies several agents learning in a shared environment, and it breaks an assumption every method so far has relied on: that the environment is fixed. When other agents are also learning, the world each one faces changes as its neighbours improve, so a policy that worked yesterday may fail today against nothing but better opponents.

The shape of the problem depends on how interests align. Cooperative MARL has agents sharing a goal, where the difficulty is coordination and deciding which agent's contribution earned a shared reward. Competitive MARL sets them against one another, where one's gain is another's loss.

Two techniques recur. Self-Play trains an agent against copies of itself, so the opposition strengthens exactly as the agent does and provides a continuously appropriate challenge — the method behind the best-known results in competitive games. Centralized Training Decentralized Execution exploits a practical asymmetry: during training all agents' information can be pooled, since training happens in a controlled setting, while each agent must ultimately act on what it alone can observe. Training with more information than execution allows turns out to make coordination markedly easier to learn.

Exploration vs Exploitation

An agent that only ever takes the action it currently believes is best will never discover a better one. An agent that only ever experiments will never profit from what it has learned. Every reinforcement learning method must resolve this tension, and the way it does so often matters more to its success than the learning rule itself.

The Exploration-Exploitation Trade-off names the dilemma. Exploiting means using current knowledge to take the highest-valued action; exploring means deliberately trying something else to improve that knowledge. The difficulty is that the value of exploring is invisible in the moment — it pays only later, and only sometimes — while the cost is immediate and certain. This is why an agent left to act greedily from the start reliably settles on a mediocre habit: having found something that works, it never gathers the evidence that would show it what it is missing.

The simplest remedy is Epsilon-Greedy: take the best known action most of the time, but with a small fixed probability pick one at random instead. It is crude — the random choice is as likely to try a clearly terrible action as a promising one — yet it is enough to keep learning alive, and it remains widely used because it is trivial to implement and hard to get badly wrong.

Its obvious flaw is that the exploration never stops: a fully-trained agent still throws away a fraction of its actions on random choices. Decaying Epsilon fixes this by shrinking that probability over time, exploring heavily when the agent knows nothing and settling toward pure exploitation as its estimates mature. The schedule matters, and it echoes the learning-rate schedules of supervised training: decay too fast and the agent commits to early impressions, too slow and it wastes its run.

Softmax (Boltzmann) Exploration addresses the other flaw — that random exploration ignores what the agent already believes. Rather than choosing uniformly among alternatives, it selects in proportion to estimated value, so a close runner-up is tried often and a hopeless action is tried rarely. A temperature setting controls how sharply it discriminates, interpolating between near-random and near-greedy behaviour.

A more principled family starts from a different question: not how often to explore, but what is worth exploring. Optimism Under Uncertainty is the guiding idea — when an option's value is unknown, assume it is good. The effect is self-correcting and elegant. An optimistic estimate draws the agent to try that option; trying it replaces the assumption with evidence; if the option was genuinely poor its estimate falls and the agent stops choosing it. Exploration becomes a consequence of how uncertainty is represented rather than a random override bolted on top.

UCB (Upper Confidence Bound) makes that concrete by adding a bonus to each action's estimated value, sized according to how uncertain that estimate is — an action tried rarely carries a large bonus, one tried often a small one. The agent then acts greedily on value-plus-bonus, so it explores exactly where its knowledge is thinnest and stops as confidence accumulates. Thompson Sampling reaches a similar end by a different route: it maintains a probability distribution over each action's true value, draws a sample from each, and acts on the samples. Actions the agent is unsure about have wide distributions and so occasionally sample high, which is what gets them tried.

The methods above all explore within a known set of choices, which is not enough when reward is rare. If an agent must act for thousands of steps before any feedback, no amount of clever action selection will help — nothing distinguishes good behaviour from bad until far too late. Intrinsic Motivation answers this by giving the agent its own internally generated reward alongside whatever the environment provides, so it has a reason to act purposefully in the meantime.

Curiosity-Driven Exploration is the best-known form, rewarding the agent for encountering situations it finds surprising — states its own predictions handle poorly. Since surprise falls as a region becomes familiar, the reward naturally pushes the agent outward into genuinely new territory rather than letting it linger. This is what allows progress in environments where the external signal alone would leave an agent wandering indefinitely.

Multi-Armed Bandits

Strip reinforcement learning down to its smallest interesting case and you get the bandit problem. There is still a choice to make, still uncertain reward, still the tension between using what you know and finding out more — but no state, and therefore no consequences that carry forward. That simplification is what makes the setting valuable: exploration can be studied on its own, with results precise enough to prove rather than merely observe.

The Bandit Problem is the canonical statement. An agent faces several options, each paying out according to its own unknown distribution, and must choose repeatedly to accumulate as much reward as it can. The name comes from slot machines — several levers, no idea which pays best, and only pulling them will tell you. Compared with a full decision process, one thing is missing and it is the decisive one: an action does not change the situation. Each round begins exactly as the last did, so the only thing carried forward is what the agent has learned.

Because the setting is so clean, success can be measured exactly. Regret is the standard measure: the difference between the reward actually collected and what would have been collected by always choosing the best option, had it been known from the start. It reframes the goal usefully — not maximising reward in the abstract, but minimising what you lose while finding out. Every round spent on an inferior option adds to it, so regret directly prices the cost of exploring, and the natural question becomes how slowly it can be made to grow.

The answer depends on how the rewards behave. In a Stochastic Bandit, each option pays from a fixed but unknown probability distribution, so the observed average converges toward a true value with enough samples. This is the setting where the sharpest results hold, and UCB1 is its standard algorithm — a concrete instance of the confidence-bound idea, adding a precisely specified uncertainty bonus to each option's running average and always picking the highest total. Its appeal is that its regret can be bounded mathematically rather than merely measured empirically.

An Adversarial Bandit abandons the assumption of fixed distributions entirely and allows payoffs to be chosen by an opponent who may react to the agent's behaviour. No amount of averaging helps here, since there is no stable quantity to converge on, and any deterministic rule can be exploited by an adversary who anticipates it. EXP3 is the standard response, maintaining weights over the options and sampling from them randomly so that its choices cannot be predicted, while still shifting weight toward whatever has been paying.

The Contextual Bandit moves back toward the full problem by giving the agent a piece of side information before each choice — a visitor's characteristics before selecting which article to show them. The best option now depends on that context, so the agent must learn a rule mapping context to choice rather than a single ranking. This is the form most often deployed in practice, in recommendation and online advertising, and it sits precisely between the plain bandit and a full decision process: the choice is informed by the situation, but it still does not change what comes next.

Tabular Methods

Before value functions were approximated by neural networks, they were simply stored — one number per state, or one per state-action pair, in a table. That restricts these methods to problems small enough to enumerate, but it is also why they are the right place to learn the subject: every idea in deep reinforcement learning appears here first, in a form where it can be understood exactly rather than approximately.

Dynamic Programming

The starting point assumes something later methods give up: that the agent knows the environment's dynamics completely. With that knowledge the Bellman equations can be solved directly, and the result is a pair of operations that alternate.

Policy Evaluation answers the first question — given a fixed policy, what is each state worth? It sweeps the state space repeatedly, updating each state's value from its successors, until the numbers stop moving. Policy Improvement answers the second — given those values, can the policy be bettered? It can, by acting greedily with respect to them, and the guarantee that this never makes the policy worse is what makes the whole scheme work.

Alternating the two is Policy Iteration: evaluate to convergence, improve, repeat. It reaches the optimal policy in surprisingly few rounds, but each evaluation phase is expensive. Value Iteration cuts that cost by refusing to evaluate fully — it performs a single sweep before improving again, folding both steps into one update.

The two are ends of a spectrum rather than rivals, and Generalized Policy Iteration is the name for the whole spectrum: any scheme in which evaluation and improvement interleave, in any proportion, converges on the same place. This is the idea worth carrying forward — almost every method in the rest of this chapter, however elaborate, is an instance of it.

Monte Carlo Methods

Dynamic programming needs a model of the environment. Monte Carlo methods need only experience: play an episode to its end, observe the return that actually followed each state, and average those returns over many episodes. Monte Carlo Prediction is that idea applied to evaluation, and it is unbiased by construction — it estimates the expected return using nothing but observed returns.

A subtlety arises when a state is visited more than once in one episode. First-Visit MC averages only the return following the first visit, while Every-Visit MC counts each visit separately. Both converge to the correct value, by slightly different arguments.

Monte Carlo Control extends this to improving behaviour rather than just evaluating it, following the same evaluate-and-improve cycle. That raises a problem the model-based case never had: a greedy policy may simply never try some actions, so their values are never learned and they never become attractive. MC Exploring Starts is the blunt fix — begin episodes from randomly chosen state-action pairs, guaranteeing every pair is eventually sampled — though it demands a control over the environment that real problems rarely allow.

A better answer is to learn about one policy while following another, and Importance Sampling (RL) makes that sound: it reweights each observed return by how likely the target policy was to have produced it. The mathematics is exact, but the weights can vary enormously across long episodes, which is why off-policy Monte Carlo can be correct and still unusable in practice.

Temporal Difference Learning

Monte Carlo waits for an episode to end before learning anything. Temporal Difference (TD) Learning refuses to wait: it updates after every single step, using its own estimate of what follows in place of the return it has not yet seen. This combination — learn from raw experience like Monte Carlo, but update incrementally like dynamic programming — is the central idea of the field.

The mechanism that makes it possible is Bootstrapping: updating an estimate from other estimates rather than from final outcomes. It sounds circular and is not, because each estimate is anchored by real observed rewards. It also introduces bias, which is the price paid for not waiting.

The quantity driving each update is the TD Error — the gap between what was predicted and the reward-plus-next-estimate actually encountered. It is worth recognising on sight, because it recurs throughout the deep methods later in this chapter as the fundamental learning signal. The simplest form, updating from a single step ahead, is written TD.

Between one-step updates and full-episode returns lies a spectrum. n-Step TD looks n steps ahead before bootstrapping, trading bias for variance as n grows — one step is maximally biased, a full episode maximally variable, and something in between usually learns fastest.

Rather than choosing one n, TD(Lambda) averages over all of them at once, weighted geometrically by a parameter. Implementing that literally would need the future, so it is achieved instead by the Eligibility Trace: a short-term memory marking how recently each state was visited, so one error can update many past states in proportion to their responsibility. It is an elegant answer to credit assignment, and it works online, one step at a time.

Tabular Control Algorithms

Applied to control, temporal-difference learning produces the algorithms most people meet first. Q-Learning is the best known: it learns the value of acting optimally regardless of how the agent is actually behaving, which makes it off-policy, and its convergence guarantee under mild conditions made it the field's workhorse for decades.

SARSA is its on-policy twin, and the difference is one term in the update — it bootstraps from the action actually taken next rather than the best available one. The consequence is behavioural: SARSA learns the value of the policy it is really following, exploration included, so near a cliff edge it learns a cautious route while Q-learning learns the optimal one and falls off it while exploring.

Expected SARSA removes the noise of that sampled next action by averaging over all of them under the current policy, which lowers variance at slightly more computation. SARSA(Lambda) adds eligibility traces to the on-policy update, propagating each error back across recently visited pairs.

One defect deserves its own name. Taking a maximum over noisy estimates systematically produces an answer that is too high, because the maximum selects whichever estimate happens to be over-optimistic — this is Maximization Bias, and Q-learning has it by construction. Double Q-Learning corrects it by keeping two independent value tables and using one to choose the best action while the other supplies its value, so the noise that inflated the choice does not also inflate the estimate. The same fix reappears later against the deep version of the same problem.

Function Approximation

Tabular methods keep one entry per state, which works only while the states can be counted. Give an agent a camera, or a handful of continuous sensors, and the table becomes impossible — and useless besides, since no state would ever be visited twice. This section covers the bridge between the tabular theory and everything built on neural networks.

Function Approximation in RL replaces the lookup table with a learned function: instead of storing a value for every state, the agent fits a function that computes one from the state's features. The table stops being a record and becomes a model, with a fixed set of parameters no matter how large the problem grows.

The point of doing this is Generalization in RL — the property that learning about one situation informs judgement about similar ones the agent has never encountered. In a table there is no such transfer: two nearly identical states are unrelated entries. This is not merely an efficiency gain but the only way an agent can act sensibly in a world it cannot exhaustively visit, and it is why approximation is a necessity rather than a compromise.

The simplest form is Linear Function Approximation, where value is a weighted sum of features. It is well understood theoretically, its convergence properties are provable in cases where the nonlinear version's are not, and it remains a sound choice when good features are available. That condition is the catch, and it is what Feature Engineering for RL addresses: with a linear model, everything depends on describing the state in terms that make the value function easy to express. Getting this right was the bulk of the work before deep networks began learning features for themselves.

Two classical schemes for building those features are worth knowing. Tile Coding overlays several offset grids on the state space and represents a state by which cell it occupies in each — cheap to compute, and giving fine resolution from coarse tiles because the offsets between grids break ties. Radial Basis Function (RL) features instead measure closeness to a set of reference points, producing smoothly varying activations rather than the on-or-off pattern of tiles.

Approximation also introduces a genuine danger, and it has a name: the Deadly Triad. Three ingredients are each individually reasonable — function approximation, bootstrapping, and off-policy learning — and combining all three can cause value estimates to diverge without bound rather than converge. What makes this so consequential is that the three together describe exactly the most useful methods, deep Q-learning included. The instability is not hypothetical, and the replay buffers, target networks and constrained updates of the deep algorithms are best read as accumulated defences against it.

Deep Reinforcement Learning

Tabular methods store one number per state, which confines them to problems small enough to enumerate — and almost nothing interesting is. Replacing that table with a neural network lifts the ceiling entirely, letting an agent learn from raw pixels or continuous sensor readings. It also destroys the convergence guarantees the tabular theory provided, and most of what follows is the accumulated engineering that makes the combination work anyway.

Value-Based Deep RL

The breakthrough was the Deep Q-Network (DQN), which learned to play Atari games from screen pixels using the same architecture across dozens of titles. Naively bolting a network onto Q-learning diverges, and two additions are what stopped it.

Experience Replay stores past transitions in a buffer and trains on random samples from it, rather than on each experience as it arrives. This breaks the strong correlation between consecutive frames — which otherwise violates the assumptions the network's optimiser relies on — and lets each experience be reused many times. The Target Network addresses the other instability: the target being chased is computed from the very network being updated, so it shifts with every step. Freezing a copy for a period gives the updates a stationary target to aim at.

The refinements that followed each target a specific flaw. Double DQN carries the tabular maximization-bias fix into the deep setting, decoupling action selection from value estimation. Dueling DQN splits the network into two streams, one estimating the state's overall worth and one the relative merit of each action, which helps where most actions make little difference. Prioritized Experience Replay stops sampling the buffer uniformly and draws surprising transitions — those with large errors — more often, on the reasoning that there is more to learn from them. Rainbow DQN combines these and several other improvements in one agent, and its result was the useful finding that the gains are largely complementary rather than redundant.

Policy Gradient Methods

Value-based methods derive behaviour from estimated values, which requires a maximum over actions — awkward when actions are continuous, and incapable of representing a deliberately random policy. Policy Gradient Methods take the direct route and adjust the policy's parameters to increase expected return.

That this is even possible rests on the Policy Gradient Theorem, which gives the gradient of expected return in a form that can be estimated from sampled experience without differentiating through the environment's unknown dynamics. REINFORCE is the simplest algorithm built on it: play an episode, then push up the probability of the actions taken in proportion to the return that followed. It is correct and unbiased, and its variance is severe enough to make it slow.

Reducing that variance is what the rest of this section is about. A Baseline (Policy Gradient) subtracts a reference value from the return before weighting, so actions are judged against what was expected rather than in absolute terms — this changes nothing in expectation while cutting variance sharply. Using the state's value as that reference yields the Advantage Function, which measures how much better an action was than the state's average. It is one of the most reused quantities in modern reinforcement learning.

Two further refinements change the step itself. Natural Policy Gradient measures distance between policies by how much their behaviour differs rather than by how much their parameters differ, which makes progress independent of how the policy happens to be parameterised. Deterministic Policy Gradient (DPG) handles continuous control by learning a policy that outputs a single action rather than a distribution, avoiding the intractable integral a stochastic policy would require.

Actor-Critic

The two families above are complementary, and Actor-Critic combines them: an actor holds the policy and chooses actions, while a critic learns a value function and judges them. The actor gets the low-variance feedback it needs without waiting for episodes to end, and the critic supplies exactly the baseline the previous section wanted.

Advantage Actor-Critic (A2C) is the standard form, with the critic estimating the advantage directly. Generalized Advantage Estimation (GAE) refines how that advantage is computed, blending estimates over many horizons with a single parameter that trades bias against variance — the same spectrum eligibility traces walk in the tabular case, applied here. Asynchronous Advantage Actor-Critic (A3C) attacks the problem from the systems side, running many actors in parallel on separate environment copies so their combined experience is diverse enough to decorrelate updates without a replay buffer at all.

Trust Region & Modern Algorithms

Policy gradients have a characteristic failure: one oversized update can collapse a policy that took hours to train, and because the next batch of data is gathered by that ruined policy, there is no path back. The algorithms here are the field's answer, and they are what practitioners actually reach for.

TRPO states the fix rigorously, constraining each update so the new policy cannot differ too much from the old one, with a monotonic-improvement argument behind it. It is effective and computationally heavy. PPO achieves nearly the same effect by simply clipping the update when the policy ratio strays too far — vastly simpler to implement, robust across a wide range of tasks, and consequently the default choice for most work today, including the alignment of language models.

Continuous control has its own lineage. DDPG combines deterministic policy gradients with the replay buffer and target network of value-based deep RL, making it off-policy and sample-efficient though notoriously sensitive to settings. TD3 stabilises it with three specific corrections, the central one being a second critic to counter the same overestimation bias seen throughout this chapter. SAC (Soft Actor-Critic) takes a different angle by rewarding the policy for remaining random as well as for earning return, which keeps exploration alive of its own accord and yields notably robust performance.

Two entries generalise beyond single algorithms. IMPALA is a distributed architecture for training at scale across many machines, correcting for the lag between the actors gathering experience and the learner consuming it. Distributional RL changes the target itself: rather than learning the expected return, it learns the whole distribution of possible returns, which turns out to improve performance even when only the mean is ultimately used.

Model-Based RL

Most methods in this chapter learn purely from trial and error, never forming any picture of how the world works — they discover which actions pay without ever predicting what those actions will cause. This section covers the alternative, which is also the more obviously intelligent one: learn how the environment behaves, then use that knowledge to think ahead before acting.

Model-Based RL names the approach. Its appeal is sample efficiency: a model can be practised against indefinitely without touching the real environment, which matters enormously when real interaction is slow, expensive or dangerous — a robot that can rehearse internally needs far fewer real attempts. The cost is that errors in the model compound, and a policy optimised against a flawed model can exploit its mistakes, performing beautifully in imagination and failing in reality.

The learned model is often called a World Model — a representation predicting how the environment evolves in response to actions. Planning with Learned Models is what makes it useful: simulating candidate courses of action forward and choosing on the results, rather than relying only on remembered values.

Dyna-Q is the classic demonstration that the two approaches need not compete. It learns a model from real experience and then trains its value estimates on both real transitions and simulated ones drawn from that model, so each real interaction is amplified into many updates. The architecture is simple and the lesson general: model-free learning and model-based planning can share a single value function.

Monte Carlo Tree Search (MCTS) is the planning algorithm that matters most here. It builds a search tree by repeatedly simulating to a leaf, expanding it, playing out or evaluating from there, and propagating the outcome back — concentrating effort on the promising branches rather than searching uniformly. It handles enormous branching factors precisely because it never commits to exhaustive search.

The best-known results in the field come from combining that search with learned networks. AlphaGo paired it with networks trained on human games and self-play to defeat a world champion at Go. AlphaZero removed the human data entirely, learning from self-play alone and reaching stronger play across Go, chess and shogi with one method. MuZero went further still and removed the rules: rather than being given the environment's dynamics, it learns a model of just those aspects that matter for predicting value and reward, which lets the same approach work where the rules are unknown — Atari games as readily as board games.

Reward Engineering & Imitation

Every method in this chapter maximises a reward signal, and each is judged on how well it does so. This section asks the question those methods take for granted: where does the reward come from? It is the point at which reinforcement learning stops being a mathematical problem and becomes a design one, because an agent optimising the wrong objective will pursue it with exactly the same competence it would have brought to the right one.

Reward Design

Reward Engineering is the craft of turning what you actually want into a number the agent can maximise. It is harder than it appears, because the reward must be honest about the goal while being learnable in reasonable time — and those two pull in opposite directions. A reward given only at success is unambiguous but so sparse the agent may never stumble on it.

Reward Shaping is the usual remedy: add intermediate rewards that guide the agent toward the goal, so it receives feedback before achieving anything final. Done carefully this speeds learning enormously. Done carelessly it changes the problem, because the agent optimises the shaped signal rather than the intent behind it — reward a robot for approaching a door and it may learn to hover near the door rather than pass through it.

That failure has a general form. Reward Misspecification is the gap between the reward written down and the outcome intended, and it is the ordinary case rather than an exotic one, since any short numerical proxy for a rich goal will omit something. Reward Hacking is what an agent does with that gap: it finds behaviour scoring highly under the stated reward while defeating its purpose entirely — circling a target to collect points instead of finishing a race, or exploiting a physics bug to gain height. These are not malfunctions. The agent is doing precisely what was asked, which is what makes the problem hard, and the reason careful reward design matters as much as any algorithm here.

Imitation Learning

An alternative sidesteps reward specification: if a competent demonstrator exists, learn from them instead. Imitation Learning covers this family, and it is often the practical answer where a reward would be almost impossible to write down but a human can readily show the behaviour.

The simplest version is Behavioural Cloning: treat the demonstrations as labelled data and train a policy to reproduce the expert's action for each observed state. This is plain supervised learning, and it works until the agent makes a small error. Because that error moves it somewhere the expert never visited, its next decision is made on unfamiliar ground, producing a larger error still — mistakes compound rather than correct, and the agent drifts away from anything it was taught.

DAgger addresses that directly by letting the agent act, then asking the expert what they would have done in the states the agent actually reached. Training on that data covers recovery from the agent's own mistakes, which the original demonstrations never contained.

A deeper approach infers the objective rather than copying the behaviour. Inverse Reinforcement Learning (IRL) takes demonstrations and recovers a reward function that would explain them, on the reasoning that the expert's goal generalises to new situations while their specific actions do not. It is genuinely underdetermined — many rewards explain the same behaviour — but a recovered reward can then be optimised by any method in this chapter. GAIL (Generative Adversarial Imitation Learning) reaches a similar end by adversarial training: a discriminator learns to tell agent behaviour from expert behaviour, and the agent is trained to fool it, matching the expert's distribution without ever recovering an explicit reward.

Extensions & Variants

The methods earlier in this chapter share a set of assumptions: the agent learns one task, from scratch, by interacting freely with its environment, and nothing terrible happens while it experiments. Each of those assumptions fails somewhere in practice, and this section collects the research directions that relax them.

Long tasks strain flat decision-making, because a policy choosing one primitive action at a time must connect a reward to a decision made thousands of steps earlier. Hierarchical RL introduces levels: a high-level policy selects sub-goals while low-level policies carry them out, so credit travels across a handful of decisions rather than thousands. The Options Framework is its standard formalisation, defining a temporally extended action with its own initiation condition, internal policy, and termination rule — an action that takes however long it takes, which lets the standard theory apply unchanged at the higher level.

Two ideas concern the order and reuse of learning. Curriculum Learning presents tasks in increasing difficulty, so that skills acquired on easy cases make hard ones reachable — problems that are hopeless when attempted directly often become straightforward when approached through a graded sequence. Transfer Learning in RL reuses what was learned on one task to accelerate another, which matters because training from scratch is the field's dominant cost.

Meta-Reinforcement Learning pushes that further, aiming to learn the act of learning: an agent trained across many related tasks acquires a strategy for adapting quickly to a new one, so what transfers is not a policy but the ability to acquire a policy from a few episodes.

A different assumption fails when interaction itself is unavailable. Offline RL learns from a fixed dataset of previously collected experience with no further environment access at all — the setting whenever exploration is dangerous or expensive, as in medicine or industrial control. Its central difficulty is that the agent cannot test its ideas: a policy that favours actions the dataset barely covers will have wildly overconfident value estimates and no way to discover otherwise. Off-Policy Evaluation is the necessary companion, estimating how a proposed policy would perform using only logged data, since deploying it to find out is exactly what the setting forbids.

Safe RL takes on the assumption that experimentation is harmless, adding constraints the agent must respect while learning rather than merely optimising return — indispensable anywhere a mistake damages equipment or people, and where the ordinary licence to try everything cannot be granted.

Finally, RLHF (RL from Human Feedback) addresses the case where the objective cannot be written down at all. Rather than specifying a reward, humans compare pairs of outputs; a reward model is trained to predict those preferences, and the policy is optimised against it. This is how large language models are aligned to be helpful and harmless, and it is currently the most economically significant application of anything in this chapter.

RL Frameworks & Environments

Reinforcement learning depends on running an agent against something, over and over, for millions of steps. That makes tooling unusually consequential here: the standard environments are what allow two published results to be compared at all, and the standard libraries are what stop every project reimplementing algorithms whose details are notoriously easy to get subtly wrong.

OpenAI Gym mattered less for any environment it shipped than for the interface it established — reset to begin, step with an action, receive an observation, a reward and a signal that the episode has ended. Once every environment presented that shape, an algorithm could be written once and pointed at anything, and the field gained a common language it had lacked. Gymnasium is its maintained successor, carrying the same interface forward with corrections; it is what current work should target, and the older name persists mostly in existing code and papers.

Several environment suites became benchmarks in their own right. The Atari Learning Environment supplies dozens of games sharing one observation format — raw pixels — and one control scheme, which is what let a single agent be evaluated across many tasks without per-game engineering, and it is the benchmark on which deep reinforcement learning first proved itself. MuJoCo serves the opposite need: a fast, accurate physics simulator whose continuous-control tasks are the standard proving ground for the policy-gradient and actor-critic algorithms, where the difficulty lies in smooth coordinated motion rather than in perception.

Two others target richer settings. DeepMind Lab offers three-dimensional navigation and puzzle tasks from a first-person view, where the agent sees only part of its world and must build understanding over time. Unity ML-Agents takes a different route by turning a general-purpose game engine into a training platform, so anyone able to build a scene can create a custom environment — valuable precisely because the standard benchmarks resemble few real problems.

On the algorithm side, two libraries dominate. Stable Baselines3 provides careful, well-tested implementations of the main algorithms and is the usual starting point: reinforcement learning results are famously sensitive to implementation details that papers omit, so a reference implementation is worth more here than in most fields. RLlib addresses scale instead, offering distributed training across many machines for problems where a single process cannot gather experience fast enough.

Applications

Reinforcement learning fits a particular shape of problem: a sequence of decisions, where each choice changes the situation the next one faces, and where success is measured over the long run rather than at any single step. Where that shape holds and interaction is cheap enough to gather experience, the results have been striking. Where it does not, or where mistakes during learning are costly, adoption has been slower — and the gap between those two conditions explains most of the pattern below.

Game AI (Chess, Go, Atari) is where the field's landmark results came from, and the reason is not that games matter but that they are ideal: rules are exact, a simulator runs millions of games overnight, and the score is unambiguous. Every one of those conditions is a luxury elsewhere, which is worth remembering when reading such results as evidence of general capability.

Robotics Control is the natural physical counterpart — continuous control over many timesteps, with a goal easier to demonstrate than to describe. It is also where the difficulties are sharpest: real robots move slowly, wear out, and cannot afford the failures learning requires, so training typically happens in simulation and must then survive the gap to reality. Autonomous Driving presses that harder still, since the sequential structure fits perfectly while the safety constraint forbids the exploration these methods depend on, which is why the technique tends to appear in components and simulation rather than in end-to-end control.

Infrastructure has proved more tractable, because the systems are already instrumented and already simulated. Adaptive Traffic Signal Control replaces fixed timings with policies responding to live demand, where the sequential effect is direct — holding one light changes the queues every downstream junction inherits. Energy Grid Optimization balances generation, storage and demand over time, a problem defined by decisions whose consequences arrive hours later.

The same reasoning extends into operations. Supply Chain Optimization concerns ordering and inventory under uncertain demand, where each decision alters the position the next is made from, and Resource Allocation (RL) covers the general problem of assigning limited capacity — compute, staff, bandwidth — as conditions shift. Portfolio Optimization applies it to allocating capital over time, with the caveat that financial markets violate the usual assumption of a fixed environment: other participants adapt, and a strategy learned on history need not survive contact with a market that has moved on.

Recommendation Systems (RL) closes the list and is among the most widely deployed. Framing recommendation as a sequential problem rather than a series of independent predictions captures something real — what a system shows now shapes what a user does next, and optimising each click in isolation tends to narrow the experience over time. The contextual bandit setting from earlier in this chapter is the form most often used in practice.