Time Step

Reinforcement learning unfolds as a sequence of small interaction beats. A time step is one such beat: the agent observes where it is, chooses an action, and receives the environment’s response before the next decision begins.

What happens in one step

At time step t, the agent has an observation or state, written s_t, and selects an action a_t using its policy. The environment then advances, producing a reward r_t+1 and a new observation s_t+1. This is the basic transition:

observe s_t
act with a_t
receive reward r_t+1 and next state s_t+1

The subscript matters: it records order, not wall-clock time. A step might represent 20 milliseconds in a robot simulator, one move in chess, or one day in an inventory-control system.

Why the count matters

Time steps connect immediate choices to delayed consequences. An RL agent does not judge an action only by its next reward; it estimates the rewards that follow across later steps. The discount factor γ controls how strongly rewards far ahead in this sequence count. In an episode, the step count also determines when a horizon is reached or when the environment declares terminated or truncated.

In practice

In a Gymnasium environment, calling env.step(action) advances exactly one time step. A DQN implementation stores each resulting transition (s, a, r, s′) in replay memory; PPO collects many consecutive steps before updating its policy. Choosing the wrong step duration can break the task: a drone controlled too infrequently cannot correct its motion, while a trading agent whose “step” is a day cannot react to intraday events. Clear time-step boundaries make rewards, value estimates, and credit assignment refer to the same interaction rhythm.

A time step is one discrete iteration of the reinforcement-learning interaction loop: the agent observes a state, selects an action, and receives a reward and the next observation from the environment. Indexed by t, it orders a trajectory and defines when rewards, transitions, and policy decisions occur. Time steps matter because return calculations, discounting, credit assignment, and value updates all depend on their temporal sequence.

Think of playing a board game one turn at a time. You look at the board, make a move, then see what changes. Each turn is a small, clear moment in the game.

In reinforcement learning, a time step is one such moment. The learning system observes its situation, chooses an action, and then receives feedback—perhaps a reward, a penalty, or simply a new situation to deal with. For a robot, one time step might be taking a single step forward. For a game-playing AI, it might be one move. Breaking experience into time steps lets the system connect its choices with what happens next.