Curiosity-Driven Exploration

When an agent receives a reward only after a rare success, waiting for external reward can leave it wandering almost blindly. Curiosity-driven exploration gives the agent an additional internal reason to investigate: it rewards experiences that are new, surprising, or useful for improving its understanding of the environment.

How the internal reward works
The agent learns from a combined reward: external reward from the task, plus an intrinsic reward generated from its own learning process. A common design gives a bonus when the agent cannot yet predict the consequences of an action. As its world model improves, familiar transitions become less rewarding internally, encouraging it to move on.

Prediction-error curiosity: reward the mismatch between predicted and observed next states.
Novelty-based curiosity: reward states the agent has visited rarely, or states that look unlike its stored experience.
Information-gain curiosity: reward actions that reduce uncertainty in the agent’s model.

Why it helps—and where it fails
In a sparse-reward maze, a curiosity-driven agent can be motivated to open doors, enter unexplored rooms, and eventually discover the goal; a purely reward-driven agent might repeat unproductive actions forever. This is especially valuable in simulated robotics tasks in MuJoCo, where the task reward may appear only after a long sequence of coordinated movements. Methods such as Intrinsic Curiosity Module (ICM) add this exploratory signal to policies trained with algorithms like PPO.

The “noisy television” problem
Raw surprise is not always useful. A screen showing random static is impossible to predict, so a prediction-error agent can become fascinated by it instead of pursuing the task. Good curiosity mechanisms therefore try to reward learnable novelty: situations that are unfamiliar now but become understandable through experience. The intrinsic reward must also be scaled carefully; if it dominates the external reward, the agent becomes an explorer that never finishes its job.

Curiosity-driven exploration augments external reward with an intrinsic reward for visiting novel, surprising, or poorly predicted states. The agent is therefore motivated to seek information even when the environment provides sparse or delayed rewards. It matters because it improves discovery of useful behaviours and states that reward-only exploration might never reach, though intrinsic rewards must avoid repeatedly rewarding unpredictable noise.

Imagine exploring a new city with no guidebook. You could keep visiting the café you already know is good, or wander down an unfamiliar street just to see what is there. Curiosity-driven exploration gives a learning system that second impulse: a reason to investigate things that are new, surprising, or not yet understood.

This matters when ordinary rewards are rare. A robot might receive a reward only when it finds a distant goal, so curiosity encourages it to explore interesting places along the way. It is not curiosity in the human emotional sense. It is a practical way to help a system discover useful experiences instead of getting stuck repeating only what it already knows.