Gradient Checking

Gradient checking is a debugging tool for answering a crucial question: “Is backpropagation computing the derivatives my network actually needs?” Before trusting a model that refuses to learn, it helps separate a flawed gradient implementation from ordinary training difficulties such as a poor learning rate or insufficient data.

Two ways to measure a slope
Backpropagation calculates an analytic gradient: the derivative of the loss with respect to every parameter, obtained efficiently with the chain rule. Gradient checking compares it with a numerical gradient, estimated by slightly nudging one parameter, θ, and observing how the loss changes:

numerical_gradient ≈ [L(θ + ε) - L(θ - ε)] / (2ε)

The centered difference above is far more accurate than changing the parameter in only one direction. For a small sample of weights, compare the analytic and numerical values using relative error:

|g_analytic - g_numeric| / max(1e-8, |g_analytic| + |g_numeric|)
Errors around 1e-7 or smaller are reassuring in double precision; much larger errors point to a bug.

What it catches—and what can mislead it
Gradient checking catches reversed signs, missing chain-rule factors, incorrect broadcasting, and errors in a custom PyTorch autograd.Function. It is especially valuable after implementing a new layer or loss from scratch. Check a tiny network and a few randomly chosen parameters: checking every weight is expensive because each numerical estimate requires extra forward passes.

Use a controlled test
The comparison only works when the loss is deterministic. Disable dropout, freeze batch-normalization statistics, use fixed inputs and targets, and avoid data augmentation. Nondifferentiable points—such as a ReLU exactly at zero—can legitimately disagree with a finite-difference estimate. A clean gradient check does not guarantee a model will train well, but a failed one means optimisation is built on unreliable directions; no choice of Adam settings can repair that.

Gradient checking validates gradients computed by backpropagation by comparing them with numerical estimates obtained from small perturbations of each parameter, typically using finite differences. It is a debugging tool for verifying loss functions and derivative implementations, not a training method. Correct gradients are essential: errors in them cause optimisation to stall, diverge, or update parameters in the wrong direction.

Imagine checking a new kitchen scale: you place a known weight on it to see whether its reading is trustworthy. Gradient checking is a similar safety check for an AI system while it is learning.

A learning network needs to know which tiny adjustments will improve its answers. It calculates these directions using gradients—signals that say “change this a little up” or “a little down.” Gradient checking compares those calculated signals with a slower, independent estimate. If they disagree, it can reveal a bug in the learning setup.

It is mainly used during development, not normal training: a careful test that helps make sure the AI is learning for the right reasons.