Upstream Gradient

Picture a neural network as a long chain of small calculations. During training, each calculation needs to hear how its output affected the final mistake; that incoming message is the upstream gradient.

The message arriving from the loss
For an operation that transforms an input x into an output y = f(x), the upstream gradient is the derivative of the loss with respect to that output: ∂L/∂y. It arrives from the parts of the network closer to the loss during the backward pass. The operation combines it with its own local derivative, ∂y/∂x, to produce the gradient for its input:

∂L/∂x = (∂L/∂y) × (∂y/∂x)

This is the chain rule in action. For vector-valued layers, the multiplication is performed with a Jacobian—or, in efficient implementations, a Jacobian-vector product—rather than forming a huge derivative matrix explicitly.

How gradients travel through a network
A linear layer receives an upstream gradient at its output and uses it to compute gradients for both its weights and its input. An activation such as ReLU gates that message: inputs that were negative have a local derivative of zero, so no gradient passes through them. At a branch, such as a ResNet skip connection, gradients from every route back to a shared earlier value are added together. This accumulation is why skip connections create reliable paths for learning.

Why the quality of this signal matters
Every layer transforms the upstream gradient before passing it backward. Repeated multiplication by small derivatives makes it vanish; repeated multiplication by large values makes it explode. Early layers then learn painfully slowly or updates become unstable and the loss diverges. LayerNorm, careful initialization, residual connections, and optimizers such as Adam help keep this signal useful. In PyTorch, autograd constructs and applies these backward computations automatically, but inspecting parameter gradients still reveals whether the upstream signal is reaching each layer at a healthy scale.