DDPG
DDPG, short for Deep Deterministic Policy Gradient, is designed for control problems where an agent must choose precise continuous actions: how far to turn a steering wheel, how much torque to apply, or where a robot joint should move. Rather than selecting from a fixed menu of actions, it directly outputs a number or vector of numbers.
How it learns
DDPG is an off-policy actor–critic algorithm built from two cooperating neural networks:
- The actor maps a state to one deterministic action, such as a steering angle.
- The critic estimates Q(s, a): the long-term reward expected after taking that action in that state.
The critic learns from transitions stored in a replay buffer, using a Bellman target based on its estimate of the next state. The actor then adjusts its output in the direction the critic says has higher value. This lets DDPG handle actions that would be impractical to enumerate, unlike DQN’s discrete-action setup.
Stability and exploration
Deep RL can become unstable when a network learns from targets produced by another rapidly changing network. DDPG addresses this with slowly updated target networks for both actor and critic, making training targets move gradually. Because its policy is deterministic, it also needs deliberately injected exploration noise during data collection; without it, a robotic arm could repeat an early mediocre movement forever and never discover a better one.
Practical strengths and pitfalls
DDPG was influential for continuous-control benchmarks such as MuJoCo, and appears in implementations derived from libraries like Stable Baselines3. Its main weakness is that critic errors can be amplified: an actor seeks actions the critic rates highly, including actions rated highly only because of estimation error. This can produce brittle policies that look successful in training yet fail after a small change in dynamics. TD3 improves this design by using two critics and delayed policy updates, reducing that overestimation problem.
DDPG (Deep Deterministic Policy Gradient) is an off-policy actor–critic algorithm for continuous-action reinforcement learning. It learns a deterministic policy that selects actions directly and a critic that estimates their long-term value, using replay buffers and slowly updated target networks to stabilize neural-network training. DDPG enables efficient learning in high-dimensional control tasks, but its sensitivity to exploration noise and value-estimation errors motivated successors such as TD3.
Imagine teaching a robot arm to place a glass on a table. It can move left, right, up, down, and anywhere in between—not just choose from a few preset moves. DDPG, short for Deep Deterministic Policy Gradient, is a method for learning in these smooth, continuous-action situations.
It learns by trying movements, seeing whether they help, and gradually favouring the moves that lead to better results. “Deterministic” means that, once it has learned, it usually picks one specific best movement for a situation. This makes DDPG useful for tasks like steering a car, controlling a robot, or adjusting the power of a machine.