Cross-Attention
Cross-attention is how one set of tokens looks up useful information in another set. It gives a model a controlled bridge between two representations: one side asks questions, while the other side supplies the material from which answers are assembled.
How the lookup works
In attention, each token is projected into a query, key, and value. With cross-attention, queries come from one sequence or stream, while keys and values come from a different one. The model computes a similarity score between each query and every key, scales and normalizes those scores with softmax, then takes a weighted mixture of the values. A query therefore receives information mainly from the external tokens judged most relevant to it.
Encoder–decoder example
In the original Transformer for translation, the decoder is generating one output token at a time. Its current hidden states provide the queries; the encoder’s representations of the input sentence provide keys and values. Cross-attention lets the decoder focus on the relevant input words while producing each output word. Unlike self-attention, where a stream attends within itself, cross-attention connects distinct streams. In practice, multi-head attention runs several such lookups in parallel, allowing different heads to learn different relationships.
Why it matters in training
Cross-attention creates a direct, differentiable route for information and gradients between components. It is central when a model must condition one representation on another, such as decoder states on encoder states or learned queries on an external feature sequence. Its main cost is compute and memory: comparing every query with every key requires an attention-score matrix whose size grows with both sequence lengths. Poor masks can expose padding or forbidden positions; mismatched dimensions prevent the projections from aligning; and unstable attention scores can make the softmax overly sharp or diffuse. Standard scaled dot-product attention, residual connections, and normalization keep this block trainable inside deep Transformer stacks.
Cross-attention is an attention mechanism in which queries come from one representation stream while keys and values come from another. Each query computes relevance-weighted access to information in the other stream, allowing one set of tokens or features to condition on another. It matters because it enables controlled information exchange between distinct sequences or representations, such as an encoder’s outputs and a decoder’s current states.
Imagine a translator reading a French sentence while writing its English version. For each new English word, they look back at the French words most relevant right now. Cross-attention gives an AI a similar ability: while working on one set of information, it can selectively consult another.
For example, an image-generating system uses the words “a red bicycle” to guide which parts of an emerging picture it should focus on. A captioning system can look at regions of a photo while choosing each word. This matters because the AI does not treat all outside information as equally useful. It can connect the right clue from one source to the task happening in another.