Encoder Block

An encoder block is a reusable transformer component that lets each input token refine its representation by looking at other tokens in the same input. Stack many such blocks, and the network builds increasingly useful internal descriptions: early blocks capture direct relationships, while later ones combine them into broader context.

What happens inside the block
A standard encoder block has two main transformation stages, each protected by a residual connection and normalisation:

Multi-head self-attention lets every token assign different weights to every other token. Each attention head learns a distinct kind of relationship, then their outputs are combined.
A position-wise feed-forward network applies the same small MLP to every token independently. Attention mixes information between tokens; this network transforms the newly mixed features.

In a common “pre-norm” design, such as modern transformer implementations, the flow is roughly: normalise → attention → add the original input; then normalise → feed-forward network → add the intermediate result. Positional information must be supplied separately, because self-attention alone has no built-in sense of order.

Why residuals and normalisation matter
The residual path gives information and gradients a direct route through depth. Rather than forcing each block to rebuild a representation from scratch, the block learns a correction to what is already present. LayerNorm keeps activation scales controlled for each token, making training less sensitive to depth and learning rate. Without these safeguards, a deep stack can develop unstable activations, fading gradients, or a loss curve that suddenly diverges after a few epochs.

Training behavior in practice
Encoder blocks are stacked in models such as BERT and in the encoder side of the original Transformer. Dropout is commonly applied to attention outputs and feed-forward outputs during training, reducing reliance on a few fragile pathways; it is disabled during inference. The block’s cost is dominated by self-attention: comparing all token pairs requires memory and compute that grow roughly with the square of sequence length. Frameworks expose this directly—for example, PyTorch’s TransformerEncoderLayer packages attention, feed-forward layers, normalisation, residual additions, and dropout into one trainable unit.

An encoder block is a Transformer layer that updates each token representation using self-attention, a position-wise feed-forward network, residual connections, and layer normalization. Self-attention mixes information across tokens, while the feed-forward network transforms each token independently. Stacking encoder blocks builds increasingly contextual representations; residual paths and normalization keep gradients stable and enable deep Transformer training.

Imagine a group of readers examining the same sentence together. Each reader can look at every word, notice which words seem connected, and update their understanding based on the whole sentence. An encoder block is one such round of group reading inside a transformer.

It takes a set of input pieces—such as words in a sentence—and turns them into richer descriptions that include context. For example, it can help distinguish “bank” beside “river” from “bank” beside “money.” Stacking many encoder blocks lets the AI build increasingly useful understanding, from nearby word relationships to the broader meaning of a passage.