Partially Observable MDP (POMDP)

A robot driving through fog does not see the full road situation; it sees a noisy, incomplete slice of it. A Partially Observable Markov Decision Process (POMDP) describes this kind of reinforcement-learning problem: the true situation exists, but the agent must act without directly observing all of it.

Hidden state and useful memory
In a standard MDP, the current observation is treated as the complete state: it contains everything needed to predict what actions will lead to. In a POMDP, the environment has a hidden state, while the agent receives an observation generated from that state. The same observation can therefore call for different actions depending on what happened earlier.

A card-game agent sees its own cards but not its opponent’s hand.
A navigation robot’s camera view cannot reveal whether it previously turned left or right.
A trading agent sees prices, not the underlying market conditions that produced them.

Acting from beliefs rather than certainty
The agent handles missing information by using its action-observation history to construct a belief state: a probability distribution over plausible hidden states. For example, a robot hearing a sound behind a closed door might assign some probability to “person,” “machine,” and “nothing,” then choose whether to investigate. An agent can also learn a compact memory directly, such as a recurrent neural network inside a policy. Recurrent PPO and recurrent variants of DQN use this approach, carrying information forward across time rather than treating each observation independently.

Why partial observability changes learning
A memoryless policy can score well in training yet fail when visually identical situations require different responses. Partial observability also makes information-gathering actions valuable: looking around, checking a sensor, or briefly exploring can improve later decisions even when it gives no immediate reward. If the agent’s memory, belief update, or training sequence length is misconfigured, its value estimates mix together distinct hidden situations and its behaviour becomes unreliable. POMDPs capture a central reality of reward-driven learning: agents must decide not only what to do, but also what they need to find out before doing it.

A Partially Observable Markov Decision Process (POMDP) models decision-making when an agent cannot directly observe the true environment state. Instead, it receives incomplete or noisy observations and must use its action–observation history, or a belief over possible states, to choose actions. POMDPs matter because many real RL tasks require memory and information gathering; treating them as fully observable can produce unstable or systematically poor policies.

Imagine driving through thick fog. You cannot see the whole road, nearby hazards, or what lies around the next bend. You make choices using partial clues: road markings, headlights, sounds, and what you remember from moments earlier.

A Partially Observable MDP, or POMDP, describes this kind of decision problem for an AI. The AI must act without seeing the full situation. It may need to remember past clues, make sensible guesses about what is hidden, and sometimes choose actions that reveal more information. This matters because many real settings—robots in buildings, medical decisions, and card games—never provide a complete picture.