Padding Mask

Neural networks prefer neatly rectangular batches, but real sequences rarely have the same length. A padding mask lets a transformer keep that convenient rectangular shape without mistaking the extra filler positions for real data.

What the mask does
Shorter sequences in a batch are extended with a special padding token until they match the longest sequence. During attention, each query produces scores against every key. The padding mask marks which key positions contain padding, and the model replaces their attention scores with a very large negative value before the softmax. After softmax, those positions receive essentially zero attention weight, so their value vectors contribute nothing to the attention output.

Why this is necessary
Without masking, padding is not harmless empty space. Even a learned padding embedding can produce keys and values that attract attention. A short sequence could then devote part of its attention budget to meaningless filler, altering hidden representations and gradients. The model could also learn accidental correlations with sequence length or with where padding begins. In a PyTorch MultiheadAttention layer, this role is commonly supplied through key_padding_mask; transformer APIs may instead call it attention_mask or provide it alongside a separate causal mask.

Padding versus causal masking
These masks solve different constraints:

A padding mask blocks positions that do not belong to the example, regardless of their location.
A causal mask blocks future positions, preserving left-to-right prediction.
Decoder-style models using padded batches generally need both: no attention to padding, and no peeking ahead.

A common bug is reversing the mask convention: one library treats 1 as “keep,” while another treats True as “ignore.” The result can be loss curves that stall or behave erratically because the model is attending to filler—or masking the actual sequence. Padding masks add negligible compute compared with attention itself, while making batched training valid.

A padding mask marks artificial padding tokens added to make sequences in a batch the same length. In attention, it suppresses scores involving those positions—typically by assigning them a large negative value before softmax—so they receive no attention weight. This prevents meaningless padding from altering token representations, attention distributions, gradients, or loss calculations during batched training.

Imagine a teacher grading worksheets of different lengths. To stack them neatly, the teacher adds blank pages to the shorter ones—but knows to ignore those blanks when reading answers.

A padding mask plays that role in an AI language system. Sentences in a batch are often padded with empty placeholder slots so they all have the same length. The mask tells the system, “These slots are not real words. Do not pay attention to them.”

Without it, the model could treat meaningless blanks as part of the sentence and make worse predictions. The padding mask keeps its attention focused only on genuine content.