Position-wise Feed-Forward Network

A transformer needs two complementary kinds of work: deciding which positions should exchange information, and transforming the information held at each position. The position-wise feed-forward network handles the second job: it is a small neural network applied independently to every token or position.

How the block works
After self-attention produces a vector for each position, the same two-layer network is run on each vector separately. Conceptually, for an input vector x, it computes a linear expansion, applies a nonlinearity such as GELU, then projects back to the model’s original width:

FFN(x) = W₂ · GELU(W₁x + b₁) + b₂.

The hidden layer is commonly four times wider than the model dimension. For a 768-dimensional transformer representation, the FFN might expand it to 3,072 dimensions before compressing it again. Every position uses identical weights, but each position’s values produce its own result. This is why it is “position-wise”: unlike attention, it does not directly mix information between positions.

Why transformers need it
Attention is excellent at routing information—connecting a position to relevant positions—but its main operation is weighted averaging. The FFN supplies the nonlinear, feature-building capacity that lets a transformer reinterpret what attention has gathered. Together, the two parts divide the work:

Self-attention mixes information across positions.
The FFN mixes and transforms features within each position’s vector.

In a standard transformer block, the FFN sits inside a residual connection and near LayerNorm, which keeps deep stacks trainable. Its wide intermediate activations consume substantial memory and compute; in many transformer variants, the FFN holds more parameters than attention. Dropout can be applied within it during training and is disabled during inference. If this network is too narrow, the model lacks transformation capacity; if its activations, initialization, or learning rate are unstable, loss can spike or training can plateau. Modern models also replace the basic GELU FFN with gated versions such as SwiGLU for stronger capacity at similar cost.

A Position-wise Feed-Forward Network (FFN) is a small multilayer perceptron applied independently and identically to each token representation in a transformer, typically using two linear layers separated by a nonlinear activation. It expands and transforms each token’s features without mixing information across positions. FFNs supply much of a transformer’s per-token nonlinear modeling capacity; without them, attention alone provides limited feature transformation.

Imagine a group of students discussing a story, then each student spending a moment with the same private tutor. The group discussion helps everyone hear useful context; the tutor then helps each person sharpen their own understanding.

A position-wise feed-forward network plays that private-tutor role in a transformer. It takes each word or piece of text individually and enriches its representation—its internal “meaning note”—in the same consistent way. It does not decide which words should talk to each other; another part handles that. Its job is to give every position a chance to process and refine what it has learned from the surrounding context.