Value Vector

A value vector is the information a transformer actually carries forward after attention decides what to look at. If attention is a librarian: queries state what is needed, keys label each book, and values contain the passages copied into the answer.

For every input position, a transformer creates three learned projections of its representation: a query (Q), a key (K), and a value (V). A query is compared with every allowed key using a dot product. After scaling and a softmax, those comparison scores become attention weights. The output is a weighted sum of value vectors:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V

Keys determine where attention goes; values determine what content comes back. A position with a high attention weight contributes much more of its value vector to the output.

Keeping keys and values separate lets the model learn different representations for matching and for communication. A token can expose one set of features to help other tokens find it through its key, while storing different useful features in its value. In multi-head attention, each head has its own value projection, so different heads can collect different kinds of information from the same positions. The heads’ outputs are concatenated and mixed by another learned linear layer.

Value vectors are learned through backpropagation along with the query and key projections. Poor value representations limit the information attention can transmit even when the model attends to the right places. Conversely, good values are wasted when query–key scores route attention incorrectly. In a PyTorch attention module, the value projection is commonly a learned Linear layer producing a tensor shaped like [batch, heads, positions, value_dim]. Attention’s major cost comes from comparing every query with every key; values add the weighted aggregation step. Causal or padding masks change which keys can receive weight, and therefore which value vectors are allowed into each output.