Inverse Reinforcement Learning (IRL)

Imagine watching an expert navigate a crowded warehouse. You can see which turns they take, but nobody has written down the trade-offs in their head: speed versus safety, short routes versus smooth ones, avoiding blocked aisles versus staying near a charging station. Inverse Reinforcement Learning (IRL) tries to infer those hidden preferences from demonstrated behaviour.

From actions back to objectives
In ordinary reinforcement learning, a designer supplies a reward function, and an agent learns a policy that earns high reward. IRL reverses that direction: it receives expert trajectories—sequences of states and actions—and searches for a reward under which those choices would be sensible. It then commonly runs a standard RL planner on the inferred reward to produce a policy.

Why inference is not simple copying
The same behaviour can fit many rewards. An expert who takes a wide turn might value safety, dislike sharp steering, anticipate a future obstacle, or simply have imperfect information. IRL therefore needs assumptions to choose among plausible explanations, such as preferring simpler rewards or modelling the expert as approximately optimal. A common workflow is:

• represent possible rewards using features of states or actions;
• compare expert trajectories with trajectories produced by a policy under a candidate reward;
• adjust the reward until expert-like behaviour is assigned higher value or probability;
• optimize that reward, including in states the demonstrations did not explicitly cover.

Why it matters in practice
IRL addresses a painful RL problem: a hand-written reward can accidentally reward a shortcut rather than the real goal. A robot rewarded only for reaching a location might rush through unsafe gaps; demonstrations can reveal that experts preserve clearance and move smoothly. Unlike behavioural cloning, which directly imitates observed actions and can drift after one unfamiliar state, IRL aims to recover a transferable objective. Its limits are equally important: poor or narrow demonstrations teach poor preferences, and an incorrect environment model can make a recovered reward look convincing while producing unsafe plans. Maximum Entropy IRL handles demonstrator variability by treating good actions as likely rather than requiring perfect optimality; it helped inspire methods such as GAIL.