Notes
🔍

Forward Pass

A neural network’s forward pass is its trip from input to answer. Data enters the network, each layer transforms it using its current parameters, and the final layer produces a prediction such as a class score, probability, or numeric value.

What travels through the network
In a simple dense layer, the incoming values x are multiplied by learned weights W, shifted by a learned bias b, and passed through an activation function: z = Wx + b, then a = f(z). The resulting activations become the next layer’s input. A forward pass therefore consists of repeated transformations:

  • Input features enter the first layer.
  • Hidden layers compute intermediate representations.
  • Activations such as ReLU introduce nonlinearity, allowing the network to represent more than one large linear formula.
  • The output layer produces predictions in the form required by the task.

Its role in training
During training, the forward pass also computes the prediction used by the loss function. The loss measures the gap between prediction and target; then the backward pass uses stored intermediate values to calculate gradients and update weights. This storage is why training consumes much more memory than inference. In PyTorch, calling a module invokes its forward() method; autograd records the operations when gradients are enabled.

Training mode versus inference mode
The same network can behave differently during its forward pass depending on its mode. Dropout randomly removes some activations during training but is disabled during inference. BatchNorm uses mini-batch statistics while training and saved running statistics when evaluating. Forgetting model.eval() can therefore make evaluation predictions noisy or inconsistent. In a ResNet, skip connections also participate in the forward pass by adding an earlier activation to a later one, giving information—and later, gradients—a more direct route through depth. A broken forward-pass shape, unstable activation scale, or incorrect training/evaluation mode can make a model fail before optimisation has a fair chance to help.

The forward pass is the computation that moves an input through a neural network layer by layer, applying learned weights, biases, activations, and any architectural operations to produce an output or prediction. It defines the model’s current mapping from inputs to outputs. During training, its intermediate values and loss provide the quantities that backpropagation uses to compute gradients and update parameters.

Imagine sending a package through a series of sorting stations. At each station, the package is read, sorted, and passed to the next one. By the end, it reaches its destination.

A neural network’s forward pass is similar. It takes an input—such as a photo, a sentence, or a sound clip—and moves it through its layers in one direction. Each layer picks out useful clues: perhaps edges in a photo, then shapes, then a face. The final layer produces the network’s answer, such as “this is a cat.”

The forward pass is the network’s moment of making a prediction from what it currently knows.