TD3

TD3 is a practical answer to a frustrating deep-RL problem: an agent can become highly confident in actions that only look good because its value estimates are wrong. It is especially useful for continuous controls, such as choosing a robot’s joint torques or a vehicle’s steering angle.

TD3, short for Twin Delayed Deep Deterministic Policy Gradient, is an off-policy actor–critic algorithm built on DDPG. Its actor network outputs one deterministic continuous action for each state. Two critic networks estimate the action’s expected long-term reward, using transitions stored in a replay buffer. TD3 then adds three safeguards:

• Clipped double Q-learning: it trains two critics and uses the smaller target estimate. This deliberately cautious choice counters the optimism that can cause an actor to exploit a critic’s mistakes.
• Delayed policy updates: critics are updated more frequently than the actor, so the actor is not constantly chasing a moving, unreliable value signal.
• Target-policy smoothing: small clipped noise is added to the target action when computing critic targets. The critic learns that nearly identical actions should have similar values, rather than rewarding a brittle, narrow spike.

In a MuJoCo locomotion task, a DDPG agent might discover that its critic assigns an implausibly high value to one precise torque pattern. The actor then repeats and amplifies that error, producing unstable learning or a policy that collapses under a tiny change in dynamics. TD3’s minimum of two critic estimates makes this shortcut harder to exploit, while smoothing discourages dependence on exact actions that work only in simulation. Because it reuses past experience, TD3 can be sample-efficient; because its policy is deterministic, exploration still requires deliberately adding noise during data collection. Libraries such as Stable Baselines3 provide TD3 for continuous-action environments, where these stability measures made it a strong baseline before entropy-based methods such as SAC became widely used.