State Space

A robot navigating a room, a game-playing agent viewing a board, and a thermostat reading a building’s temperature all need a way to describe “where things stand right now.” The state space is the set of all such situations the reinforcement-learning problem allows.

In a Markov decision process, a state contains the information needed to predict the consequences of an action: the next state and reward distribution depend on the current state and chosen action, not on an earlier hidden history. The state space, usually written S, is the collection of every possible state.

• For Gridworld, it might be every square the agent can occupy.
• For chess, it is the enormous set of legal board configurations, including whose turn it is.
• For a MuJoCo robot, it can be a continuous vector of joint positions, velocities, and relevant object locations.

An RL policy chooses actions from states, and value functions estimate expected future return from them: V(s) asks how promising state s is, while Q(s,a) asks how promising action a is there. A poorly chosen state space makes these estimates unreliable. For example, a driving agent given position but not speed cannot tell whether braking will stop it safely. Two physically different situations have been collapsed into the same input, breaking the Markov property and forcing the agent to guess from incomplete evidence.

The true state need not be directly visible. A poker player cannot see opponents’ cards; a robot camera cannot directly observe hidden objects or friction. The agent receives an observation instead. When observations omit decision-relevant facts, the problem is partially observable, and the policy may need recent history or a recurrent network to construct an internal belief about the state. In practice, environments such as Gymnasium expose an observation space, which is the agent’s input interface—not automatically a complete Markov state.