Discount Factor (Gamma)

Imagine choosing between a small reward now and a larger reward later. The discount factor, written γ (gamma), is the setting that tells a reinforcement-learning agent how seriously to take that “later” reward. It sets the agent’s time perspective: impatient, long-sighted, or somewhere between.

How gamma changes the objective

At time t, an agent aims to maximize its discounted return:

Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + ...

Rewards one step away are multiplied by γ, two steps away by γ², and so on. In the usual continuing-task setting, 0 ≤ γ < 1. A gamma of 0 means “care only about the next reward.” A gamma near 1 gives substantial weight to distant consequences. For example:

γ = 0.9: a reward 10 steps away counts as about 35% of its original size.
γ = 0.99: that same reward still counts as about 90%, supporting much longer-term planning.

Why it matters in learning

Gamma is not merely a preference knob. It also makes the learning target mathematically manageable: in continuing environments, discounting keeps an infinite stream of rewards from producing an unbounded return and helps Bellman updates settle toward a stable value. In DQN, for instance, the target contains r + γ max Q(s', a'); gamma determines how much the value estimate at the next state influences the current one.

Practical trade-offs

A low gamma makes credit assignment easier because the agent focuses on nearby effects, but it can miss strategies whose payoff arrives late—such as taking a temporary detour to reach a valuable goal. A very high gamma captures those long-term consequences, yet value estimates must propagate reward across many steps and can become slower or less stable to learn. In a finite Gymnasium episode, termination also matters: once the task truly ends, there are no future rewards to discount. Choosing gamma therefore defines what “good behavior” means across time, not just how an algorithm is tuned.

The discount factor, denoted γ (gamma), is a value between 0 and 1 that weights future rewards when computing an agent’s return: rewards received later count less than immediate rewards. A larger γ makes the objective more long-term; a smaller γ emphasizes short-term outcomes. It defines the effective planning horizon and helps keep infinite-horizon returns finite and stable.

Imagine choosing between a cookie now and a cake promised tomorrow. Most people value the cookie a little more because it is certain and immediate. A discount factor, often called gamma, gives an AI a similar sense of how much to care about rewards that arrive later.

If gamma is high, the learner is patient: it will accept a small inconvenience now for a bigger payoff in the future, like taking a longer route to avoid traffic. If gamma is low, it focuses more on immediate rewards. This matters because good decisions often have delayed consequences, while far-off outcomes are usually less certain.