GAIL (Generative Adversarial Imitation Learning)

Imagine learning to drive by watching skilled drivers, without anyone having to write down a reward for every good lane change, brake, and turn. GAIL turns that idea into a reinforcement-learning procedure: it learns a policy whose behaviour is difficult to distinguish from expert demonstrations.

GAIL, short for Generative Adversarial Imitation Learning, trains two models against each other. A policy produces state–action pairs by interacting with the environment; a discriminator sees pairs from both the expert and the policy and tries to tell their source apart. The policy receives a learned reward for fooling the discriminator, then improves itself with an RL algorithm such as TRPO or PPO. At success, the policy visits and acts in situations with a similar distribution to the expert—called matching occupancy measures.

This avoids designing a hand-written reward, which can be surprisingly dangerous. For example, a robot rewarded only for moving forward could exploit a simulator bug, fall in a way that registers progress, or race unstably toward the goal. With demonstrations, GAIL instead learns the behavioural pattern that experts exhibited, including choices that are hard to encode explicitly.

• Unlike behavioural cloning, it learns through environment interaction, so it can recover from states caused by its own small mistakes.
• Unlike full inverse reinforcement learning, it does not first recover an explicit human-interpretable reward function.
• It is useful for tasks such as locomotion in MuJoCo, where expert trajectories can define “natural” movement better than a simple reward formula.

GAIL still needs rollouts in the environment, making it expensive when simulation is slow or real-world trials are unsafe. Its discriminator can overfit a small demonstration set, producing a misleading reward; unstable adversarial training can also leave the policy with weak or noisy learning signals. Crucially, it cannot reliably learn behaviour absent from the demonstrations: an expert who never encounters a slippery floor or unusual obstacle gives the learner little guidance there. GAIL therefore replaces reward engineering with a new requirement: broad, high-quality examples of the behaviour worth imitating.