Horizon
When an agent acts, it needs some notion of how far ahead its choices should count. The horizon is that planning window: the number of future steps whose rewards are included when judging an action.
What the horizon controls
In a finite-horizon problem, an episode lasts exactly H steps. At time t, the agent seeks reward through the remaining H − t steps. A chess-like task with a fixed move limit, a robot assigned a ten-second maneuver, or a Gymnasium episode capped at 500 steps all have an explicit horizon. The value of acting from state s is written conceptually as the expected sum of rewards until that endpoint.
Infinite tasks and discounting
Some environments have no natural endpoint: controlling a data centre, managing inventory, or balancing a robot indefinitely. These are treated as infinite-horizon problems. Rather than adding an endless sequence of rewards directly, RL commonly uses a discount factor, γ, making rewards further in the future count less:
return = rₜ₊₁ + γrₜ₊₂ + γ²rₜ₊₃ + ...A smaller γ creates a shorter effective horizon: the agent focuses on near-term consequences. A γ close to 1 asks it to care far ahead. This is not merely a mathematical convenience; it changes behaviour. An agent trained to collect immediate points can discover a reward-function loophole, while one with a longer horizon can accept a short-term cost to reach a genuinely better future state.
Why it matters in practice
Horizon determines how difficult credit assignment becomes. Long horizons make it harder to identify which early action caused a reward much later, and value estimates in algorithms such as DQN or PPO become more sensitive to approximation error. A time limit can also silently distort learning: ending a MuJoCo rollout at step 1,000 is different from the environment truly reaching a terminal state. Treating that cutoff as failure incorrectly tells the agent that no future value exists beyond it. Choosing the horizon and handling termination correctly therefore defines what “good long-term behaviour” actually means.
Horizon is the number of future time steps over which an agent’s rewards and consequences are considered when evaluating actions. In a finite-horizon problem, an episode ends after a fixed number of steps; an infinite-horizon problem continues indefinitely, typically using discounting. The horizon determines how far ahead an RL agent must plan and how difficult long-delayed credit assignment becomes.
Think of planning a road trip: you might choose a route differently if you are driving only to the next town than if you are travelling for a whole week. The horizon is that planning distance: how far into the future a learning system treats consequences as relevant.
A short horizon means it mainly cares about what happens soon, like a robot grabbing the nearest object. A long horizon means it also considers later effects, such as whether taking a shortcut now leaves it stuck later. This matters because good decisions often involve giving up a small immediate reward for a better outcome down the road.