Fully Connected Layer

A fully connected layer is the neural-network version of letting every incoming signal have a say in every outgoing decision. It takes a vector of numbers, mixes them in learned proportions, and produces a new vector that can represent a more useful pattern for the next layer.

What it computes
Also called a dense layer, it connects each input feature to every output unit. For an input vector x, its calculation is:

y = Wx + b

Here, W is a matrix of learned weights and b is a learned bias vector. If the layer receives 128 values and produces 64, it contains 128 × 64 weights plus 64 biases. Each output unit forms a weighted combination of all 128 inputs. A nonlinear activation function, such as ReLU or GELU, usually follows; without it, multiple fully connected layers would collapse into one larger linear transformation.

How it fits into a network
A stack of fully connected layers forms a multilayer perceptron (MLP). During the forward pass, each layer transforms the previous layer’s activations. During backpropagation, gradients flow through every connection, telling each weight how to change to reduce loss. In PyTorch, this layer is written as nn.Linear(in_features, out_features).

• In a hidden block, it expands, compresses, or remixes learned features.
• At an output, it can turn hidden features into class scores, a numerical prediction, or parameters for another component.
• Inside a transformer, dense projections create query, key, value, and feed-forward representations.

Why the “fully” part matters
Full connectivity is flexible: the layer can learn interactions between any pair of input features. Its cost grows quickly, though. A 4,096-by-4,096 layer has more than 16 million weights, consuming memory and compute and creating many opportunities to overfit. Dropout is commonly placed after dense activations during training to discourage reliance on particular units; it is disabled at inference. Poor initialization or an excessive learning rate can make dense-layer activations and gradients explode, visible as a loss curve that suddenly diverges. Normalisation and a well-tuned optimiser such as Adam help keep these layers trainable.