State

A state is the agent’s working picture of where it is in a decision process: the information it uses to choose an action and predict what could happen next. In a game, it might include the board position and whose turn it is; for a robot, it could include joint angles, velocity, and nearby obstacles.

In the clean mathematical version of reinforcement learning, a state contains all decision-relevant information from the past. Once the current state and action are known, the earlier history should add nothing useful for predicting the next state and reward. This is the Markov property:

• the environment moves from state s to a new state s′ after an action a;
• the agent receives a reward r;
• a policy chooses actions from the current state, written as π(a | s).

This setup lets methods such as DQN learn a value for “how promising is this situation?” and lets PPO learn which action is best in that situation.

The agent does not always see the true state. A driving simulator’s true state might include every car’s position, speed, road friction, and hidden intent, while a camera supplies only pixels: an observation. A single image cannot reveal whether a car is accelerating or braking, so treating it as a complete state can lead to bad decisions. Agents address this partial information by combining recent observations, using a recurrent network, or maintaining a learned internal belief about hidden conditions.

State design determines what an agent can learn from reward. In a Gymnasium cart-pole task, position and velocity are included because position alone cannot tell whether the pole is falling left or right. Leave velocity out, and identical-looking inputs demand different actions; value estimates become inconsistent and training can stall. Conversely, including irrelevant or noisy details makes learning less data-efficient. A useful state is therefore neither a raw record of everything nor a convenient label: it is the information needed to connect an action now with rewards that arrive later.