Attention Score

An attention score is a model’s quick estimate of how relevant one piece of information is to another. When processing a token or other input position, the network asks: “Which available positions contain information useful to me right now?” The score is the raw numerical answer before those estimates are turned into usable weights.

How the score is computed
Each position produces a query vector, a key vector, and a value vector through learned linear layers. For a query q and a key k, scaled dot-product attention computes:

score(q, k) = (q · k) / √d_k

The dot product is large when query and key point in compatible directions, meaning the model has learned that they match. Dividing by √d_k prevents scores from growing excessively as vector dimensions increase. The scores for all candidate keys are then passed through softmax, producing non-negative attention weights that add to one. Those weights form a weighted average of the value vectors. Scores are therefore the evidence; attention weights are the normalized decision based on that evidence.

Masks and multiple perspectives
Before softmax, a mask can replace forbidden scores with a very negative value:

A causal mask prevents a position from attending to future positions.
A padding mask prevents attention from being wasted on empty input slots.

In multi-head attention, several separate query/key/value projections compute separate score tables. One head can learn a nearby structural relationship while another learns a long-range dependency.

Why scores matter in training
Attention scores determine where information and gradients flow inside a Transformer block. If scores become too sharp early in training, softmax concentrates nearly all weight on one position, starving alternatives of learning signal. If they are too flat, the model blends unrelated values and loses precision. The scaling factor, stable softmax implementation, and careful initialization help prevent this. In PyTorch, scaled_dot_product_attention packages these details efficiently, including masking. A loss curve that becomes unstable after attention layers are introduced can reflect exploding scores, an incorrect mask, or an excessive learning rate—not merely a bad dataset.

An attention score is the compatibility value between a query vector and a key vector, typically their scaled dot product. Scores measure how relevant each input position is to the current position; after masking and softmax normalization, they become attention weights used to combine value vectors. Accurate score computation lets transformers select useful contextual information, while poor scaling or masking produces unstable or invalid attention patterns.

Imagine reading a sentence and highlighting the words most useful for understanding the next word. In “The dog chased the ball because it rolled away,” you would pay most attention to “ball” to work out what “it” means.

An attention score is an AI’s version of that mental highlight. For each word or piece of information it is considering, the system gives a score showing how relevant it seems right now. Higher scores mean “focus here”; lower scores mean “this matters less.” This lets language models connect related ideas even when they are far apart in a sentence, helping them follow context and produce more sensible responses.