Positional Embeddings

Attention can compare any token with any other token, but by itself it has no sense of first, last, before, or after. Positional embeddings give a transformer the missing clue: where each token sits in the sequence.

How order enters attention

Without position information, self-attention treats a sequence as a bag of tokens: rearranging the inputs rearranges the outputs, but does not tell the model that their meaning changed. A basic solution assigns each position an embedding vector and adds it to the token’s embedding before it enters the first transformer block. The resulting vector carries both identity and location.

Learned absolute embeddings store one trainable vector for each position.
Sinusoidal embeddings compute fixed wave-like vectors from the position number.
Relative position methods tell attention how far apart two tokens are, rather than only where each sits globally.

Modern transformer choices

Many current models use relative schemes because “three steps earlier” is useful regardless of where it occurs. Rotary Position Embeddings (RoPE), used in architectures such as Llama, rotate query and key vectors by position-dependent angles. When attention takes their dot product, the result naturally reflects relative distance. This is more tightly integrated with attention than simply adding a position vector at the input. A causal attention mask still prevents looking ahead, but it does not supply exact distances or absolute locations; positional information does that separate job.

Why the design affects training

Positional embeddings are cheap compared with attention’s quadratic sequence-cost, yet they strongly shape what a model can learn. Missing or incorrectly assigned positions leave order-sensitive tasks ambiguous. A learned position table also has a fixed maximum length: it has no trained vector for a longer unseen position. Fixed or rotary schemes can be evaluated at longer lengths, though successful length extrapolation still depends on the training setup and position-scaling choices. In practice, a model that performs well on short sequences but fails as length grows can be revealing a positional-encoding limitation rather than a failure of the attention layers themselves.

Positional embeddings are vectors added to token representations to encode each token’s position in a sequence. Because self-attention alone is permutation-invariant, these embeddings supply order information through learned, fixed, or relative position schemes. They matter because they let a transformer distinguish sequences containing the same tokens in different orders, enabling it to model sequential structure and dependencies correctly.

Imagine reading a recipe after someone shuffled its steps. You would see all the right words—“bake,” “mix,” “serve”—but not know what comes first. A transformer has a similar problem: by itself, it can look at all the words in a sentence, but it does not automatically know their order.

Positional embeddings are like small location labels attached to each word or piece of text: “first,” “second,” “third,” and so on. They give the AI a sense of sequence, helping it distinguish “dog bites man” from “man bites dog.” This matters because word order often changes meaning completely.