Weight

A neural network learns by adjusting thousands, millions, or billions of tiny numerical settings. A weight is one of those settings: it controls how strongly one input or neuron’s output influences the next neuron.

How a weight affects a neuron
A neuron first forms a weighted sum of its inputs:

z = w₁x₁ + w₂x₂ + ... + b

Here, each x is an input, each w is a weight, b is a bias, and z is the value passed into an activation function. A positive weight makes a larger input push the neuron upward; a negative weight makes it push downward; a weight near zero makes that input matter very little. Weights are not hand-written rules. During training, the network discovers useful values from data.

How weights learn
Training compares the network’s prediction with the target using a loss function. Backpropagation calculates a gradient for every weight: a signal indicating whether increasing or decreasing that weight would reduce loss. An optimiser such as Adam then updates it. In a simple form:

w = w - learning_rate × gradient

Repeated over many examples, these small changes let early layers learn broadly useful transformations and later layers combine them into decisions. A weight is therefore both a connection strength and a piece of the model’s stored knowledge.

Why weight values matter

• Initialization matters because weights that start too large can produce exploding activations or gradients; values that are too small can make signals fade through depth.
• Learning rate controls update size. Excessively large updates make weights bounce around and loss diverge; tiny updates leave training apparently stuck.
• Regularization, such as weight decay, discourages unnecessarily large weights and helps prevent memorising training examples.
• In a ResNet, skip connections provide alternate paths for signals and gradients, making useful weight updates reach very early layers.