Bias

A bias is a small, learnable adjustment added to a neuron’s weighted input. It gives the neuron freedom to shift its response rather than being forced to react only when the weighted inputs happen to line up around zero.

How it works inside a neuron
A typical neuron first computes a pre-activation:

z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
output = activation(z)

Here, the weights decide how strongly each input matters, while bias (b) is a constant offset. During backpropagation, the optimiser learns both. Without a bias, a linear layer can only produce outputs constrained to pass through the origin: when every input is zero, its output must be zero. Adding a bias removes that unnecessary restriction.

Why the offset matters

• For a ReLU neuron, a positive bias can move its pre-activation above zero, allowing it to become active; a strongly negative bias keeps it inactive.
• In a classifier’s final layer, biases act like baseline preferences before the input-specific evidence is considered. A class that is common in the training data can acquire a larger initial logit bias.
• Biases let decision boundaries shift away from the origin. A single perceptron without bias can only draw a separating boundary through that point.

In real network layers
In PyTorch, nn.Linear and nn.Conv2d include bias parameters by default. They are tiny compared with weight tensors, so their memory and compute cost is negligible. One important exception is a layer immediately followed by BatchNorm: BatchNorm already learns a shift parameter, so the preceding layer’s bias is usually disabled (bias=False) because it is redundant. In transformer blocks, LayerNorm similarly includes a learnable offset in many implementations. Here “bias” means this trainable offset, not the separate idea of statistical or social bias.