Markov Property
Think of a good state as a compact “what matters now” snapshot. The Markov property says that, once this snapshot is known, the future does not need the full story of how the agent got there.
What the property means
Formally, after the agent takes action a in state s, the distribution of the next state and reward depends only on s and a, not on earlier states, actions, or rewards. In shorthand:
- P(s′, r | s, a, history) = P(s′, r | s, a)
This does not mean the world has no past. It means the state already carries every past-dependent detail needed to predict its consequences. In chess, the board position is close to Markov: it determines legal moves and the next position. A video frame from a driving task is not necessarily Markov, because it may hide the car’s speed, road friction, or a vehicle just outside the frame.
Why RL depends on it
The Markov property makes the core RL shortcut possible: assign a value to a state, or a state–action pair, rather than to every complete history. This supports the Bellman equation, which updates an estimate using immediate reward plus the estimated value of the next state. DQN and PPO rely on this idea when they learn from transitions such as (s, a, r, s′).
When the snapshot is incomplete
If the agent’s input is only a partial observation, treating it as a Markov state can make learning unstable or misleading. A robot trained from single camera images might choose the same steering action in two visually similar frames despite moving at different speeds. Useful fixes include:
- adding missing measurements, such as velocity;
- stacking recent observations; or
- using a recurrent policy that builds an internal memory.
These cases are modeled as partially observable MDPs: the underlying environment can be Markov even when the agent cannot directly see a sufficient state.
The Markov property states that, given the current state and action, the distribution of the next state and reward is independent of the earlier history. In reinforcement learning, it makes the current state a sufficient summary for predicting consequences and choosing actions. This property underpins Markov decision processes and enables value functions and policies to be defined over states rather than complete interaction histories.
Imagine playing a board game where, before each move, you can see the whole board. You do not need a diary of every move that led there: the board’s current layout tells you everything important for choosing what to do next.
The Markov property is this idea applied to a learning system. Its next situation and reward should depend only on what is happening now and the action it takes now—not on the full hidden history. This matters because it lets the system treat each current situation as a useful starting point for deciding what to try next, rather than needing to remember every past event.