Causal (Masked) Attention

Causal attention gives a transformer a sense of time order: when predicting the next item in a sequence, it can look backward but not forward. This prevents the model from “cheating” by reading the answer it is supposed to predict.

In ordinary self-attention, each position compares its query with every position’s key, then uses the resulting weights to combine value vectors. Causal attention inserts a mask before the softmax. For position t, all positions after t receive a score of negative infinity, so softmax assigns them exactly zero probability. The attention matrix therefore has a lower-triangular shape: token 4 can attend to tokens 1–4, never 5 onward.

This setup supports two useful modes:

• During training, the entire sequence is processed in parallel. The mask enforces the same information limits that generation will have, while the model learns to predict every next token at once.
• During inference, generation proceeds one position at a time. Implementations store past keys and values in a KV cache, avoiding repeated computation over the prefix.

For example, when predicting the fourth word of “the cat sat ___,” the model may use “the cat sat” but cannot inspect the target word or later text. This is what makes decoder-only transformers such as GPT suitable for autoregressive generation.

Without the mask, training loss can look impressively low because each position accesses future context. At generation time that context disappears, and output quality collapses. Causal masks are distinct from padding masks, which block artificial filler positions rather than future positions; models handling padded batches commonly use both. The mask adds little arithmetic cost, but long sequences still make attention expensive because every permitted query-key pair must be compared. In PyTorch, causal behavior can be requested through attention APIs with an is_causal setting or an explicit triangular mask.