Continuing Task
A continuing task is a reinforcement-learning problem with no natural finish line. The agent keeps acting, receiving rewards, and influencing what happens next indefinitely—more like controlling a building’s heating system than playing a game with a final score.
What changes when there is no terminal state
In an episodic task, such as navigating a maze, an episode ends at success, failure, or a time limit. Its return is the reward accumulated until that end. In a continuing task, the interaction is modeled as an endless sequence:
state → action → reward → next state → action → ...Because simply adding every future reward would usually produce an unbounded quantity, standard RL uses a discount factor γ, with 0 ≤ γ < 1:
Gₜ = Rₜ₊₁ + γRₜ₊₂ + γ²Rₜ₊₃ + ...Discounting makes nearer rewards count more and keeps the value target finite. Another formulation evaluates average reward per time step, useful when the long-run operating rate matters more than short-term outcomes.
What the agent is really optimizing
A continuing agent must balance immediate payoff against the future conditions it creates. For example:
- A data-center controller should save cooling energy now without gradually overheating equipment.
- A robot vacuum should keep cleaning efficiently without draining its battery in a remote corner.
- A recommender-like control system must avoid exploiting a reward proxy that looks good briefly while damaging long-run outcomes.
There is no reset to erase bad choices. A policy that repeatedly takes a small harmful shortcut can drive the system into poor states for a long time.
Why it matters in practice
The task type determines how rewards, value estimates, and evaluation are designed. Algorithms such as DQN and PPO can be used in continuing settings, but implementations commonly impose artificial rollout boundaries for batching. Those boundaries should not be mistaken for true terminal states: treating a nonterminal cutoff as terminal incorrectly removes the estimated value beyond it, biasing learning. Correct handling uses bootstrapping—estimating the remaining return from the state at the cutoff—so the agent learns the consequences of sustained behavior rather than only short fragments of it.
A continuing task is a reinforcement-learning problem with no natural terminal state: the agent interacts indefinitely and receives an ongoing stream of rewards. Its objective is therefore defined over long-run return, typically discounted return or average reward per time step, rather than total reward within an episode. This setting matters for persistent control problems, such as resource management, where policies must balance immediate rewards against sustained performance.
Think of learning to manage a home’s thermostat. There is no final “you win” moment: it keeps adjusting the heating day after day, trying to keep people comfortable while avoiding wasted energy. That is a continuing task.
In reinforcement learning, a continuing task is one where the learner keeps making decisions without a natural endpoint. A robot balancing a power grid, software managing traffic lights, or a recommendation system serving new suggestions all face this kind of ongoing job. The goal is not to finish quickly, but to make good choices consistently over time, because today’s decision can affect what happens later.