Sequence-to-Sequence

A sequence-to-sequence model turns one ordered collection of items into another. The input and output can have different lengths, and the model learns both what information to preserve and how to arrange it in the new sequence.

The encoder–decoder arrangement
In the classic design, an encoder reads the input sequence and produces contextual representations: each input position is represented with awareness of the others. A decoder then generates the output one position at a time. At every step, it uses the tokens it has already produced plus relevant encoder representations to predict the next token.

• Encoder: builds a usable internal representation of the whole input.
• Decoder: produces an output sequence autoregressively, stopping when it emits a special end token.
• Cross-attention: lets each decoder step directly inspect the most useful input positions instead of relying on one fixed-size summary.

How training differs from generation
During training, the decoder is usually given the correct previous output token, a method called teacher forcing. Its predictions are compared with the true next tokens, and gradients update both encoder and decoder. During inference, the correct next token is unavailable: the decoder must feed back its own prior prediction. This mismatch can let an early mistake influence later output. A decoder also needs a causal mask, which prevents it from seeing future target tokens during training and accidentally learning the answer in advance.

Why it matters in practice
Transformers made sequence-to-sequence modeling far more effective than early recurrent encoder–decoders. In a Transformer, cross-attention can align an output step with any input location directly, while self-attention captures relationships within each sequence. This is powerful but costly: full attention grows roughly with the product of input and output lengths for cross-attention, plus quadratic self-attention costs within long sequences. A common implementation is PyTorch’s TransformerEncoderDecoder-style structure or an encoder–decoder model such as T5. If attention masks, shifted decoder inputs, or end-token handling are wrong, training loss can look healthy while generation repeats, leaks future information, or never stops.