Scaled Dot-Product Attention

Scaled dot-product attention lets a network decide which pieces of its current input deserve the most attention. Rather than processing positions strictly one after another, it compares each position directly with every relevant position and builds a context-aware result.

How the calculation works
Each input representation is projected into three learned vectors: a query (what this position is looking for), a key (what it offers for matching), and a value (the information it can contribute). For one query, the model takes dot products with all keys. Large dot products mean a strong match. It divides these scores by √d_k, applies softmax to turn them into weights that sum to one, then takes a weighted sum of the values:

Attention(Q, K, V) = softmax((QKᵀ / √dₖ) + mask) V

Why the scaling and masks matter
If key vectors have many dimensions, unscaled dot products grow large. Softmax then becomes extremely sharp: one position receives nearly all the weight, while tiny gradients make learning unstable or slow. Dividing by √d_k keeps score magnitudes in a workable range. A causal mask sets scores for future positions to negative infinity before softmax, preventing a prediction from seeing information it should not yet know. A padding mask similarly prevents meaningless padding entries from receiving attention.

Its role inside a transformer
In a transformer block, self-attention uses the same sequence for Q, K, and V, allowing each position to gather relevant context. Multi-head attention runs several smaller attention computations in parallel, so different heads can learn different relationships. The full QKᵀ matrix costs memory and compute proportional to the square of sequence length; this is the central bottleneck for long inputs. In PyTorch, torch.nn.functional.scaled_dot_product_attention provides an optimized implementation, including masking and efficient GPU kernels.

Scaled dot-product attention computes relevance scores by taking dot products between queries and keys, dividing by the square root of the key dimension, applying softmax, and using the resulting weights to combine values. The scaling prevents large score magnitudes from saturating softmax, preserving useful gradients and stabilising transformer training.

Imagine reading a sentence and, for each word, quickly asking: “Which other words here matter most for understanding this one?” Scaled Dot-Product Attention gives a transformer that kind of focused reading ability.

For every piece of text, it compares what it is looking for with clues offered by other words, then pays more attention to the most relevant ones. In “The animal didn’t cross the road because it was tired,” attention can connect “it” to “animal.” The “scaled” part keeps those relevance choices from becoming too extreme too easily, helping the model make steadier judgments as it learns.