Notes
🔍

Decoder Block

A decoder block is the repeated processing unit that lets a transformer build an output sequence one position at a time. At each position, it combines what has already been produced with information available from an input representation, then refines that result through a small neural network.

What happens inside
In the original encoder–decoder transformer, a decoder block contains three main sublayers, each wrapped in a residual connection and LayerNorm:

  • Masked self-attention: each output token can attend only to earlier output tokens, never future ones. A causal mask sets forbidden attention scores to a very negative value before softmax, making their attention weight zero.
  • Cross-attention: the decoder’s current states act as queries, while the encoder’s output states provide keys and values. This is how the block retrieves relevant information from the source sequence.
  • Feed-forward network: the same small MLP is applied independently at every position, adding nonlinear transformation capacity after attention mixes information across positions.

Why the surrounding structure matters
A residual path gives gradients a direct route through many blocks, while LayerNorm keeps activations numerically well behaved. Modern transformers commonly use pre-normalization: normalize before each attention or feed-forward sublayer. This arrangement makes deep stacks substantially easier to train. In decoder-only models such as GPT, the cross-attention sublayer is omitted: masked self-attention and the feed-forward network are enough because there is no separate encoder to consult.

Training and generation behavior
During training, the model can process every target position in parallel because the causal mask prevents information leakage. During generation, it produces one token, feeds it back, and repeats. Recomputing attention over all previous tokens at every step would be wasteful, so implementations keep a key–value cache from earlier blocks. Without masking, training loss can look excellent while the model cheats by seeing future tokens; without stable normalization and residual paths, a deep decoder can plateau or diverge as activations and gradients become poorly scaled.

A decoder block is a transformer layer that updates token representations using masked self-attention, a feed-forward network, residual connections, and normalization. The causal mask restricts each position to information from earlier positions, enabling autoregressive prediction; in encoder–decoder models, the block can also attend to encoder outputs through cross-attention. Stacking decoder blocks builds context-aware representations for sequential generation.

Think of a decoder block as the next-word writer in a very fast collaborative writing team. It reads what has already been written, considers the relevant earlier words, and helps choose what should come next. Stack many of these blocks together, and each one adds a little more understanding of the sentence, topic, tone, and context.

In language AI, decoder blocks are what let a model produce text one piece at a time: a reply, a story, computer code, or a translation. Crucially, they only look backward at earlier text, not ahead at words that have not been generated yet. That makes their output feel like a continuing, context-aware prediction.