Discounted Return

A reward arriving now is usually more useful than the same reward arriving far in the future: it is certain, available sooner, and does not require the agent to keep succeeding along the way. Discounted return gives reinforcement learning a precise way to balance immediate payoff against later consequences.

How future rewards are counted

Starting at time step t, the discounted return is

G_t = R_(t+1) + γR_(t+2) + γ²R_(t+3) + ...

Here, γ (gamma) is the discount factor, chosen between 0 and 1. A reward one step away is multiplied by γ, two steps away by γ², and so on. With γ = 0.99, rewards far ahead still matter; with γ = 0.5, their influence fades quickly. At γ = 1, return is undiscounted total reward, which works naturally for finite episodes but can become infinite in a continuing task.

Why the discount is more than a preference

Discounting defines the objective that a policy, value function, or algorithm such as DQN tries to optimize. It also makes infinite-horizon problems mathematically manageable: the sum remains bounded when rewards are bounded, and Bellman updates become stable contraction-style targets. A small γ creates a short planning horizon, so a robot can prefer a quick nearby reward over a longer route to a much larger one. A γ too close to 1 makes distant rewards influential, but also makes credit assignment harder and value estimates more sensitive to errors.

What it changes in practice

In a Gymnasium task where an agent receives a large reward only at the finish line, discounted return lets each earlier action receive some credit for that eventual success. But the setting changes behavior:

Low γ: encourages immediate gains and can make the agent give up on delayed goals.
High γ: supports long-term planning, but can amplify bootstrapping errors when a value estimate is updated from another estimate.
Reward design matters: discounting cannot fix a reward function that accidentally pays an agent for exploiting a shortcut instead of completing the intended task.

Discounted return is therefore the ruler by which “better” behavior is measured across time, not merely a numerical trick for adding rewards.

Discounted return is the total future reward from a time step, with each reward multiplied by an exponentially decreasing factor: $G_t=\sum_{k=0}^{\infty}\gamma^k R_{t+k+1}$, where discount factor $\gamma\in[0,1)$. It defines the objective that value functions and policies optimize, balancing immediate rewards against delayed outcomes while keeping continuing tasks mathematically well-defined.

Imagine choosing between getting $10 today or a series of small payments spread over the next year. Most people value the money today a little more, because it is certain and useful right now. Discounted return gives an AI a similar way to weigh rewards over time.

It treats a reward earned soon as more valuable than the same reward earned far in the future. This helps a learning system balance long-term goals with what happens next. For example, a game-playing AI may still plan ahead to win, but it does not treat a possible reward 100 moves away as equally reliable or urgent as one it can earn now.