Attention Mechanism

Attention lets a network decide which pieces of available information matter most for its current task. Rather than compressing everything into one fixed summary, it creates a context-dependent weighted mixture: a word, position, or feature can directly draw information from the most relevant other parts.

How the computation works
Each input representation is transformed into three learned vectors: a query (what this position seeks), a key (what this position offers), and a value (the information it can contribute). For one query, the network compares it with every key using a dot product. After dividing scores by the square root of the key dimension and applying softmax, those scores become weights that sum to one. The output is the weighted sum of the corresponding values:

high query–key similarity → large attention weight;
low similarity → little contribution;
the selected values form a new, context-aware representation.

Masks and multiple perspectives
A causal mask blocks positions from attending to future positions, which is essential when predicting the next token. A padding mask prevents meaningless filler positions from influencing results. Multi-head attention runs several smaller attention calculations in parallel, each with separate learned projections. One head can learn a local relationship, while another tracks a distant dependency; their outputs are combined. In cross-attention, queries come from one stream while keys and values come from another, allowing a decoder to retrieve relevant information from an encoder.

Why it matters in training
Attention creates short, direct paths between distant positions, unlike recurrent networks, where information and gradients must pass through many sequential steps. This helps deep models preserve and use long-range relationships. In a Transformer block, attention is paired with residual connections, LayerNorm, and a feed-forward layer to keep optimisation stable. Its main cost is compute and memory: full self-attention compares every pair of positions, requiring roughly quadratic work in sequence length. Poor masking can leak future answers during training; excessive sequence length can exhaust memory; and unstable activations can make attention scores too sharp or too uniform, weakening useful routing.

Attention mechanism lets each element in a sequence compute a weighted combination of other elements’ representations. It compares a query with keys to produce relevance weights, then applies those weights to values. This enables direct, content-dependent information flow across positions, allowing networks to capture long-range dependencies and selectively focus computation on the most relevant context.

Imagine reading a long sentence and highlighting the few words that matter most for understanding the next word. An attention mechanism gives an AI a similar ability: it lets the system focus on the most relevant parts of its input at each moment.

For example, when translating “The cat sat on the mat,” the AI may pay closest attention to “cat” when choosing the matching word in another language. When answering a question about a document, it can focus on the sentence containing the answer rather than treating every sentence as equally important.

This selective focus helps modern AI handle language, images, and other complex information with much better context.