Notes
🔍

Transformer

A Transformer is a neural-network architecture built to let every item in a sequence exchange information directly with the others. Rather than passing a hidden state step by step as a recurrent network does, it builds a shared “working view” of the whole sequence at each layer, making long-range relationships much easier to learn.

Attention as the communication mechanism
Its core operation is self-attention. Each token produces three learned vectors: a query (what it is looking for), a key (what it offers), and a value (the information to pass along). Query–key similarity scores determine how strongly a token reads each value. Multi-head attention runs several such comparisons in parallel, so different heads can capture different relationships. Because attention itself has no sense of order, positional embeddings are added to token representations so the network can distinguish “first” from “last.”

What one transformer block contains
A standard block combines attention with a per-token feed-forward network:

  • Self-attention mixes information between tokens.
  • The feed-forward network transforms each token’s updated features independently, usually expanding and then shrinking its hidden width.
  • Residual connections add a block’s input back to its output, preserving a stable route for activations and gradients.
  • LayerNorm controls activation scale. Modern transformer blocks commonly use pre-normalisation: normalize before attention and before the feed-forward sublayer, which makes deep stacks train more reliably.
Dropout is active during training to reduce reliance on particular connections, but disabled at inference so predictions are deterministic for a fixed input.

Why it trains well—and where it becomes expensive
A ResNet-style residual stream, LayerNorm, and optimizers such as Adam let gradients travel through many blocks without fading or exploding. Attention also avoids the sequential bottleneck of recurrence: tokens in a layer can be processed in parallel. The trade-off is that full attention compares every pair of tokens, requiring memory and compute that grow roughly with the square of sequence length. If learning rates are too large, loss can diverge through unstable residual updates; if normalization or residual paths are misplaced, deep models can plateau because early layers receive weak, noisy gradients.

A Transformer is a neural-network architecture built from stacked blocks of self-attention, feed-forward layers, residual connections, normalization, and positional information. Self-attention lets each token directly weight information from other tokens, while the remaining components transform and stabilize these representations. Transformers enable parallel training and efficient modeling of long-range dependencies; without positional encoding, attention alone cannot distinguish token order.

Imagine reading a sentence and being able to glance at every word at once, rather than following it one word at a time. A transformer is an AI design built around that idea. When it sees “The dog chased the ball because it was moving,” it can look across the whole sentence to work out what “it” refers to.

This makes transformers especially good at handling language, where meaning often depends on words far apart. The same broad idea also helps AI work with images, sound, and other kinds of information. Many modern chatbots, translation tools, and image generators are built using transformers because they can spot important connections in large amounts of data.