Function Approximation in RL

In a tiny game, an agent can remember a separate answer for every situation it has seen. In a realistic task—driving, robotics, or a game with pixel inputs—that memory would be impossibly large. Function approximation lets the agent learn patterns that carry from familiar situations to new, similar ones.

Replacing giant lookup tables

Tabular reinforcement learning stores one value for each state or state–action pair, such as Q(s, a): the expected long-term reward after taking action a in state s. Function approximation replaces that table with a parameterised model, for example Q_θ(s, a) or a policy π_θ(a|s). Its parameters θ might belong to a linear model using hand-designed features, tile coding, radial-basis functions, or a neural network. Updating the model from one experience changes its predictions for many related states, which is the source of generalisation.

How learning works—and why it can wobble

The agent still gathers transitions: state, action, reward, and next state. It adjusts θ so its current prediction better matches a learning target. For Q-learning, that target is roughly the observed reward plus the estimated value of the best next action. This creates useful learning from limited experience, but introduces hazards:

Extrapolation error: a neural network can assign unjustifiably high value to states or actions rarely seen in its data.
Moving targets: the target contains another value estimate that changes as the network learns.
The deadly triad: combining function approximation, bootstrapping, and off-policy data can make value estimates diverge.

DQN addresses part of this instability with replay buffers and a slowly updated target network. A robot trained only in simulation can also generalise poorly when real friction or sensors differ: the approximator learned the simulator’s patterns, not necessarily the world’s.

Why it matters in RL

Function approximation makes reinforcement learning practical when states are continuous or high-dimensional, and when the agent must act in situations it has never encountered exactly before. Unlike labelled learning, the “right” output is not supplied for each input; the agent must generalise from reward-driven experience while its changing policy also changes the data it receives. The model class, features, exploration coverage, and update scheme therefore determine whether that generalisation becomes competent behaviour or unstable, confidently wrong value estimates.

Function approximation in reinforcement learning represents a value function, action-value function, or policy with a parameterized model instead of a separate table entry for every state or action. It generalizes learning from experienced situations to similar unseen ones, making RL feasible in large or continuous spaces. Its accuracy and stability directly affect policy quality; poorly controlled approximation can amplify estimation errors and destabilize learning.

Imagine learning to play a huge video game where every screen is slightly different. You could try to memorise the best move for every single situation—but there are far too many to remember. Function approximation is the shortcut: it helps a learning system spot similarities between situations and make a sensible guess about what to do in a new one.

Instead of treating each moment as completely unique, it learns broad patterns, such as “when the obstacle is close, slow down.” This matters in real-world tasks, like steering a robot or controlling a car, where possible situations are almost endless. It lets learning from past rewards carry over to unfamiliar but similar cases.