Reward Shaping

When rewards arrive only at the end of a long task, an agent can spend enormous time doing things that are neither clearly good nor clearly bad. Reward shaping adds carefully chosen intermediate feedback to make progress visible while the agent is learning.

How it works
The environment has its original reward, the signal that represents the real goal: winning a game, reaching a destination, or completing a manipulation task. Shaping adds an auxiliary reward that nudges the agent toward useful behaviour. For a robot navigating to a goal, moving closer could earn a small positive reward and moving away a small penalty. This gives the learning algorithm denser feedback than “+1 only when the robot finally arrives.”

Preserving the real objective
Naively adding bonuses can change what the agent should ultimately do. A robot might circle around collecting “move closer” points rather than finish, or a game agent might exploit an accidental scoring loophole. The best-known safeguard is potential-based reward shaping:

shaping reward = γ Φ(next state) − Φ(current state)

Here, Φ is a hand-designed estimate of how promising a state is, and γ is the discount factor. This form rewards progress between states while preserving the original task’s optimal policy under the standard RL assumptions. It changes the learning path, not the destination.

Why it matters in practice
Shaping is especially valuable for sparse-reward environments, such as a Gymnasium maze where DQN sees reward only at the exit, or a MuJoCo robot task where PPO must discover a multi-step movement before receiving success credit. Useful shaping signals include:

• reducing distance to a target,
• maintaining balance or avoiding unsafe states,
• completing meaningful subgoals.

It does not replace careful reward design: every added signal is another opportunity to reward a shortcut rather than the intended outcome. Good shaping accelerates exploration; poorly aligned shaping teaches the agent to optimise the hint instead of the task.