Skip Connection (Residual)

A very deep network can be hard to train not because its layers are incapable, but because information and learning signals must survive a long journey through them. A skip connection gives both a shortcut: it lets a layer block refine what it receives instead of forcing it to rebuild everything from scratch.

In a residual block, the input x follows two paths. One passes through ordinary layers that compute a transformation F(x; θ); the other bypasses them. Their outputs are added:

• y = F(x; θ) + x

The block therefore learns the residual: the change needed to improve its input. If no useful change is needed, it can drive F toward zero and behave close to an identity mapping. Addition requires matching tensor shapes. When the number of features or spatial resolution changes, the shortcut uses a learned projection, such as W_sx, commonly implemented with a 1×1 convolution or linear layer.

During backpropagation, the addition creates a direct route from a block’s output to its input. The gradient includes an identity contribution rather than being forced through every weight matrix and activation in the residual branch. This makes vanishing gradients far less destructive and lets networks such as ResNet train at depths that plain stacked networks struggle to reach. Transformer blocks use the same principle: each attention or feed-forward sublayer is wrapped as a residual addition, typically alongside LayerNorm.

Residual connections add little compute beyond an elementwise addition, though training still stores activations for backpropagation and projection shortcuts add parameters. They do not rescue a badly chosen learning rate: a loss curve that suddenly explodes can still signal unstable updates. But when a deep plain model plateaus because early layers receive weak gradients, residual blocks usually make optimization substantially more reliable by preserving both useful representations and a path for correction.