Backpropagation

Picture a neural network making a prediction, comparing it with the correct answer, and discovering that it was wrong. Backpropagation is the efficient bookkeeping process that tells every adjustable connection in the network how much it contributed to that error—and in which direction it should change.

How the error travels backward
A training step has two passes. In the forward pass, data moves through layers to produce an output and a loss measures its error. Backpropagation then starts at that loss and works backward through the layers. Using the calculus chain rule, it computes a gradient for each weight: the rate at which a tiny change in that weight would change the loss.

A positive gradient means increasing that parameter increases loss, so the optimiser should reduce it.
A negative gradient means increasing it reduces loss, so the optimiser should raise it.
An optimiser such as Adam combines these gradients with a learning rate and its own update rules to alter the weights.

Why depth makes it challenging
Each layer’s gradient is formed by multiplying signals passed back from later layers. Across many layers, repeated multiplication can make gradients extremely small (vanishing gradients) or huge (exploding gradients). In the first case, early layers barely update and learning stalls; in the second, weight updates become unstable and the loss can suddenly diverge. The same mechanism explains why activation functions, initialization, normalization, and architecture design matter: they shape the numbers that backpropagation must carry.

What keeps training healthy
A ResNet skip connection gives gradients a shorter route backward, helping very deep networks learn. LayerNorm, used inside transformer blocks, keeps activations in manageable ranges. Gradient clipping limits dangerously large gradients, while a lower learning rate prevents even valid gradients from causing oversized updates. Frameworks such as PyTorch perform backpropagation automatically with autograd, but they still store intermediate forward-pass values in memory because those values are needed to calculate derivatives. Backpropagation is therefore the bridge between noticing an error and making a network improve from it.

Backpropagation is the algorithm that computes how a network’s loss changes with respect to every trainable parameter by propagating gradients backward from the output through each layer using the chain rule. These gradients supply the update direction used by optimizers such as gradient descent. It makes efficient training of multilayer networks possible; unstable gradient flow can otherwise cause learning to stall or diverge.

Imagine a student taking a practice test, seeing which answers were wrong, and then tracing each mistake back to the part of their study habits that likely caused it. Backpropagation gives an AI network a similar kind of feedback.

After the network makes a prediction—such as deciding whether a photo shows a cat—it compares that prediction with the correct answer. Backpropagation sends information about the mistake backward through the network, showing each part how much it contributed. The network can then adjust itself slightly, so it is more likely to do better next time. This repeated “spot the error, learn from it” process is what lets deep-learning systems improve from examples.