Transition Function

A transition function describes what the environment does after an agent acts. It is the part of the problem that turns “I chose this action here” into “now I am in that situation,” including the uncertainty that makes real decisions interesting.

How it describes change
In a Markov decision process, the transition function is usually written as P(s' | s, a): the probability of reaching next state s' after taking action a in current state s. For every state-action pair, it assigns probabilities to possible next states, and those probabilities add to one. A deterministic environment is the simple special case: an action has exactly one next state with probability 1.

Examples and the Markov assumption
Consider a robot moving on a slippery grid:

Choosing “move right” might move it right with probability 0.8.
It might slide upward or downward with probability 0.1 each.

The transition function captures that slipperiness; the reward function separately says whether the resulting situation was good or bad. In a Gymnasium environment such as FrozenLake, these transition probabilities define the dynamics the agent must handle. The Markov assumption says that once the current state is known, the transition distribution depends on the present state and action—not on the entire earlier history. If crucial information is hidden, the apparent transition function becomes unreliable because the same observed state can lead to different outcomes for unseen reasons.

Why it matters for learning
The transition function determines how actions create future opportunities and risks. A model-based agent can learn or be given this function, then plan ahead by predicting action consequences. By contrast, DQN and many other model-free methods learn value estimates directly from sampled transitions without explicitly representing the full dynamics. Either way, the agent’s experience is generated by these transitions. If simulated dynamics are wrong—for example, a trained robot encounters slightly different friction in reality—a policy that scored well in simulation can fail abruptly. Transition dynamics are therefore the bridge between a choice now and the delayed rewards that make reinforcement learning necessary.

Transition function specifies how an environment moves from one state to the next after an agent takes an action. In a deterministic MDP, it maps a state–action pair to a single next state; in a stochastic MDP, it defines the probability distribution P(s′ | s, a) over next states. It matters because expected returns, value functions, planning, and policy consequences all depend on these state-transition dynamics.

Think of a board game: where you end up after a move depends on where you are now, what move you choose, and sometimes luck—such as drawing a surprise card. A transition function is the rulebook that describes these “what happens next” possibilities.

For a learning system, it connects a situation and an action to the next situation. Pressing left might move a robot left; driving too fast on a wet road might lead to several possible outcomes. This matters because learning by reward requires seeing how choices change the world, including when the same choice can have different results.