Local Gradient

A neural network learns by tracing responsibility backward: which internal choices nudged the final loss up or down? A local gradient is the small, immediate piece of that answer for one operation or layer—how its output changes when its own input, parameter, or activation changes.

How it works in backpropagation
Backpropagation applies the chain rule by combining an incoming, or upstream, gradient with each operation’s local gradient. For an operation y = f(x), the local gradient is dy/dx. The gradient passed to the preceding operation is:

dL/dx = (dL/dy) × (dy/dx)

Here, dL/dy says how changing y affects loss L; dy/dx says how this layer transforms a change in x. Their product says how changing x affects the loss. For a ReLU, the local gradient is 1 for positive inputs and 0 for negative ones. For a sigmoid, it is σ(x)(1−σ(x)), which becomes very small near 0 or 1.

Why local gradients control training
A deep network multiplies many local gradients along its backward path. This creates two central failure modes:

Vanishing gradients: many values below 1 shrink the signal until early layers barely update.
Exploding gradients: repeated large values make updates unstable, causing loss curves to spike or become NaN.

Activation choices, careful initialization, LayerNorm, and ResNet skip connections all help preserve useful gradient flow. A skip connection gives gradients a direct route backward, rather than forcing them through every local derivative in a long stack.

What frameworks compute
In PyTorch, each differentiable tensor operation records enough information to compute its local gradient during .backward(). For example, a linear layer computes local derivatives with respect to its inputs, weights, and bias; the optimizer such as Adam then uses the resulting weight gradients. Dropout also has a local backward rule: during training, only units kept by its random mask receive gradient; during inference, dropout is disabled. Local gradients are therefore the individual links that make—or break—the full learning signal.

Local gradient is the derivative of a layer’s output with respect to its input or parameters, computed from that layer’s own operation. During backpropagation, it is multiplied by the incoming loss gradient through the chain rule to produce gradients for earlier layers and parameter updates. Local gradients determine how each operation contributes to overall gradient flow; poorly scaled derivatives can cause vanishing or exploding gradients.

Imagine a long line of people passing feedback from a teacher back to a student. Each person can only say how much their own small action affected the final result. That personal piece of feedback is a local gradient.

In a learning network, many small units work together to make a prediction, such as deciding whether a photo shows a cat. When the prediction is wrong, the network needs to know what each unit should change. A local gradient tells one unit, “Given what came into me, how did my output influence the next step?”

These small clues are passed backward and combined, helping every part of the network learn its role.