Neural Network Basics

Every deep network, however large, is assembled from one small repeating idea: a unit that takes several numbers in, combines them into a single number, and passes it on. Once that unit is clear, and once you can see how such units are arranged into layers, the elaborate architectures later in this chapter stop looking like separate inventions and start looking like variations on one theme.

Neurons & Connections

The unit itself is the Artificial Neuron, a deliberately loose abstraction of a biological cell that keeps just one feature of it: many inputs converge on a single body, which then responds strongly or barely at all. Its direct ancestor is the Perceptron, an early single-unit model built to separate two classes. Much of what follows in deep learning can be read as an answer to what that first design could not manage on its own.

What distinguishes one neuron from another is the importance it assigns to each of its inputs. Every incoming connection carries a Weight, a number saying how much that input should count: large and positive to amplify it, close to zero to ignore it, negative to push in the opposite direction. Learning, in almost every model in this chapter, means adjusting these numbers.

Alongside its weighted inputs the neuron holds a Bias, a constant of its own that shifts the result up or down no matter what arrives. This is what allows a unit to be inherently eager or inherently reluctant, rather like a kitchen scale that reads slightly heavy whatever you place on it. Adding the weighted inputs to the bias yields one number, the Pre-Activation. It earns a name because it marks the last moment at which the neuron is still doing plain arithmetic, and it is the value every later stage of the unit begins from.

Layers & Network Topology

A single neuron is a limited thing. Useful models arrange many of them into layers, and the simplest such arrangement is the Feedforward Network, in which information travels in one direction only and never loops back on itself. Stack several layers of neurons in that manner and the result is a Multilayer Perceptron (MLP), the general-purpose architecture whose name still records its single-unit ancestor.

Layers are named by where they sit. The Input Layer is where raw values enter, with one slot for each measurement the model is given. At the far end, the Output Layer produces the answer, and its size is set by the task: one slot for a single predicted quantity, one per category when choosing among several. Everything in between is a Hidden Layer, so called because nothing outside the network ever observes its values directly. Hidden layers are where a network builds its own intermediate notions of what the input contains.

When every neuron in a layer receives a connection from every neuron in the layer before it, that layer is a Fully Connected Layer — the densest possible wiring, and the default in a plain multilayer perceptron. Pushing one example through the whole stack, layer after layer, until a prediction emerges is the Forward Pass, the operation a trained network performs every time it is used.

Stacking many layers creates a difficulty of its own: the further a signal travels through a deep stack, the more it is reshaped, and the harder it becomes for early layers to influence the result. A Skip Connection (Residual) answers this by carrying a layer's input forward and adding it back further along, giving the signal a shorter route through the network. The idea is simple and its consequences are large; it is what makes very deep stacks trainable at all, and it recurs throughout the architectures in this chapter.

Capacity & Theoretical Foundation

A natural question is what these stacks of simple units can actually represent. The Universal Approximation Theorem gives a reassuring answer: a feedforward network with even a single hidden layer can approximate an enormous range of functions to any desired accuracy, provided that layer is allowed enough neurons. What the theorem does not promise is that such a network is easy to find, or of a sensible size — it is a statement about what is possible, not about what is practical.

That gap is why the shape of a network matters. Network Depth counts how many layers are stacked between input and output, while Network Width counts how many neurons sit within a layer. The two buy different things. Depth lets a network build concepts in stages, each layer composing what the one below it produced; width lets a single stage hold more distinct pieces of information at once. In practice deep and narrow networks and shallow and wide ones with comparable size behave quite differently, which is precisely why the theorem's single-hidden-layer guarantee is not the end of the story.

Both choices feed into the Parameter Count, the total number of weights and biases a model must learn. It is the plainest measure of a network's capacity, and the one that governs how much memory it occupies and how much data it needs before its many adjustable numbers can be pinned down.

Activation Functions

A neuron's weighted sum is a straight-line operation, and stacking straight-line operations only ever produces another straight line — a hundred such layers could be collapsed into one. What rescues a deep network from that fate is the Activation Function, a small non-linear step applied to each unit's output before it travels on. It is the reason depth buys anything at all, and the choice of which one to use has quietly shaped every generation of architecture in this chapter.

The functions below are best read as a sequence of answers to one recurring problem. Early choices squashed their output into a fixed range, which turned out to stall learning in deep stacks; later ones gave up that neatness to keep signals flowing.

Saturating

The first generation squashed any input, however large, into a bounded range. The Sigmoid maps every number onto the interval between zero and one, producing a smooth S-shaped curve that reads naturally as a probability or a soft switch. The Tanh has the same S shape but spans minus one to one, so its output is centred on zero — usually the better behaved of the two when values are passed onward through many layers.

Both share a defining trait, and it is the source of the word saturating: push the input far enough in either direction and the curve flattens, so large changes in input produce almost no change in output. A flat region carries very little information backwards through the network, and in a deep stack that flatness compounds. This is the difficulty the next family was designed to escape.

Two relatives are worth separating from that story. The Hard Sigmoid replaces the smooth curve with cheap straight-line segments, trading exactness for speed where computation is tight. The Softmax is different in kind: rather than acting on each unit alone, it takes a whole layer of numbers and turns them into a set of positive values that sum to one, so they can be read together as a choice among competing categories. It belongs here by its saturating character, but it is used at the end of a network rather than between its layers.

ReLU Family

The turning point was a function almost too plain to seem worth naming. The ReLU passes positive values through untouched and replaces negatives with zero. It never flattens on the positive side, so signals keep their strength however deep the stack, and it is trivially cheap to compute. Those two properties made very deep networks practical, and it remains the default starting choice.

Its weakness is the flat half. A unit pushed permanently into the negative region outputs zero for every input it will ever see and stops adapting — it has effectively died. The rest of this family are attempts to keep the shape while removing that trap. The Leaky ReLU lets negatives through at a small fixed fraction rather than crushing them to zero, so a unit always retains a way back. The PReLU takes the same idea further by learning that fraction from the data instead of fixing it in advance.

Others reshape the negative side more substantially. The ELU curves smoothly towards a small negative floor, which pulls the average output nearer zero and tends to steady the layers above. The SELU is a carefully scaled variant designed so that, under the right conditions, the values flowing through a network keep a stable spread on their own without external help. The Softplus takes the opposite approach: a smooth curve that approximates the same overall shape while remaining gently rounded at the corner. Different again is Maxout, which does not apply a fixed formula at all but takes the largest of several learned alternatives, letting each unit shape its own response.

Smooth Modern Variants

The newest group revisits an assumption the previous family took for granted: that the corner in the curve should be sharp. Each of these keeps the essential shape while rounding that corner, and each allows slightly negative outputs near it rather than cutting them off cleanly.

The GELU weights an input by how large it is relative to the rest, producing a curve that rises smoothly rather than switching on abruptly; it has become the usual choice inside transformer models. The Swish reaches a similar shape by a different route, multiplying the input by a smooth gate derived from it, and dips slightly below zero before recovering. The Mish continues in the same direction with an even softer profile. The gains from these over a plain rectified unit are usually modest and depend on the setting, but they are consistent enough that the smooth variants now dominate the largest models.

Weight Initialization

Before a network can learn anything, every one of its weights and biases needs a starting value. It is tempting to treat this as a detail — the whole point of training is that these numbers change — but the starting point decides whether training begins at all, and for deep stacks it decides how well it proceeds. The choices below form a short history of that realisation.

The obvious first idea fails outright. Zero Initialization sets every weight to zero, which sounds neutral and is in fact fatal: if all the units in a layer start identical, they receive identical corrections and stay identical forever, so a layer of a thousand neurons learns exactly what one neuron would. The condition is called symmetry, and breaking it is the first thing an initialization scheme must do.

Random Initialization breaks that symmetry by drawing small random numbers, giving every unit a different starting point and therefore a different path. This is necessary but not sufficient, because the scale of those numbers matters enormously. Draw them too small and the signal shrinks as it passes through each successive layer until nothing measurable arrives at the far end; draw them too large and it grows instead, overwhelming the network. In a shallow model the effect is mild. In a deep one it compounds layer by layer, and the schemes that follow exist to control it.

Each sets the scale of the random draw from the size of the layer rather than leaving it to guesswork. Xavier (Glorot) Initialization chooses a spread that accounts for both the number of inputs to a layer and the number of outputs from it, aiming to keep the variance of the signal roughly steady as it travels in either direction. It was derived with the symmetric, saturating activations of the time in mind and suits them well.

He Initialization adjusts that reasoning for rectified activations. Because such a unit zeroes out roughly half of what it receives, the surviving signal is weaker than the symmetric case assumes, and He compensates with a correspondingly larger spread. It is the standard partner for the ReLU family, and pairing the wrong scheme with the wrong activation is a common and quietly damaging mistake. LeCun Initialization is the same family of reasoning scaled by the number of inputs alone, and is the scheme assumed by self-normalizing activations.

Two further schemes abandon independent random draws altogether. Orthogonal Initialization builds a weight matrix whose rows are mutually perpendicular, so the layer preserves the length of what passes through it rather than stretching or shrinking it — a property that matters most where the same weights are applied repeatedly. Identity Initialization goes further still and starts a layer as a pass-through that reproduces its input unchanged, so a freshly built network begins as something close to a shorter one and deepens as training proceeds.

Backpropagation

A network's forward direction is easy to picture: numbers enter, each layer transforms them, an answer emerges. Learning requires the opposite journey. Once the answer is wrong, every weight in the network shares some responsibility for the error, and each needs to know how much. Assigning that blame efficiently is what this section is about, and it is the single algorithm that makes training a deep model feasible.

Backpropagation works the error backwards through the network, one layer at a time, computing for each weight how a small change in it would change the final error. The essential trick is reuse: rather than examining every weight independently, which would be hopelessly expensive, it computes results for a layer once and passes them back to be reused by everything below. The cost of the backward journey ends up comparable to the forward one, and that efficiency is the whole reason deep networks can be trained.

The quantity travelling backwards is a gradient, and the term Gradient Flow describes how well it survives the trip. Healthy flow means every layer, including the earliest, receives a signal strong enough to learn from. Poor flow means the layers nearest the input barely move while those near the output adapt normally, and the depth is wasted.

At each step two pieces are combined. The Local Gradient is what a single operation contributes on its own — how its output responds to its input, considered in isolation. The Upstream Gradient is what arrives from the layers closer to the output, carrying everything the error has accumulated so far. Multiplying the two gives the gradient to pass further back. Because the journey is a long chain of such multiplications, the character of the whole depends on those factors, and that is where the trouble starts.

If the factors are consistently smaller than one, the product shrinks towards nothing over many layers. These are Vanishing Gradients: the early layers receive almost no signal, learn almost nothing, and the network behaves as though it were far shallower than it is. Saturating activations are a classic cause, since their flat regions contribute very small factors. If instead the factors are consistently larger than one, the product explodes. These are Exploding Gradients, and the symptom is the opposite: enormous updates that throw the weights far from anything useful, often producing numerical overflow within a few steps. The two failures sit at either end of one phenomenon, and much of the design in the rest of this chapter — initialization schemes, non-saturating activations, skip connections, normalization — exists to keep the chain between them.

A related complication appears whenever the same weights are applied repeatedly over a sequence. Backpropagation Through Time (BPTT) is the version of the algorithm for that setting: the repeated application is unrolled into a chain as long as the sequence, and the error is propagated back along it. Because the same weight matrix is multiplied in at every step, the shrinking and exploding tendencies above are sharply amplified, which is why sequence models needed structural remedies of their own.

Finally, an implementation of any of this is easy to get subtly wrong, and a wrong gradient does not announce itself — the network simply trains poorly. Gradient Checking is the standard defence: nudge a single parameter slightly, measure how the error actually changes, and compare that against the gradient the algorithm reported. Agreement to several decimal places is strong evidence the derivation is right. It is far too slow to use during real training, which is exactly why it is reserved for verifying an implementation once, before trusting it.

Optimizers

Backpropagation tells a network which direction each weight should move. It does not say how far, how often, or in what rhythm — and those questions turn out to matter as much as the gradient itself. This section covers the machinery that turns a stream of gradients into a trained model: the loop that drives training, the algorithms that decide each step's size, the schedules that vary that size over time, and the practical safeguards that keep long runs stable.

Training Loop & Batching

Training is a repetition, and the Training Loop is that repetition written down: feed data forward, measure the error, propagate it back, adjust the weights, repeat. Everything else in this section is a refinement of one part of that cycle.

Two units of counting run alongside it. A Training Step is one weight update — a single turn of the loop. An Epoch is one complete pass over the whole dataset, and therefore many steps. Progress is usually reported in epochs, while the mechanics of learning happen step by step.

The reason those two differ is that networks are rarely updated from the entire dataset at once, which would be slow and memory-hungry, nor from one example at a time, which is noisy and wastes modern hardware. The compromise is the Mini-Batch: a small group of examples processed together, their gradients averaged into one update. How many examples go into that group is the Batch Size, and it is one of the most consequential settings in training. Small batches give noisy gradients that often generalise well and fit in modest memory; large batches give smooth, reliable gradients and use hardware efficiently, but tend to need other settings adjusted to match.

Adaptive Optimizers

The plainest possible update multiplies every gradient by one fixed number and subtracts it. The difficulty is that a single number rarely suits every weight: some sit in steep regions and need small careful steps, others in flat regions where the same step barely moves anything. Adaptive methods address this by giving each weight its own effective step size, derived from the history of gradients that weight has seen.

Adagrad introduced the idea, accumulating the squared gradients of each weight and shrinking its step in proportion. Weights that receive large or frequent gradients are slowed; rarely-updated ones keep moving. Its flaw is that the accumulation only ever grows, so steps shrink monotonically and training eventually stalls. RMSProp fixes this by replacing the running total with a decaying average, so the distant past fades and the step size can recover. Adadelta arrives at a similar remedy while also removing the need to choose a global step size at all.

The method that became standard combines that per-weight scaling with momentum, the practice of carrying part of the previous update forward so progress accumulates along consistent directions. Adam maintains decaying averages of both the gradients and their squares, and for years it has been the default first choice for almost everything. AdamW corrects a subtle defect in how the original interacted with weight decay, separating that penalty from the adaptive scaling rather than folding it into the gradient; the change is small to describe and reliably improves generalisation, which is why it now supersedes the original in most large-scale work. Nadam is a further variant that folds a look-ahead form of momentum into the same scheme, and AMSGrad addresses a convergence flaw by preventing the per-weight scaling from ever growing back.

The remaining three are answers to scale and to cost. LAMB Optimizer was built for very large batches, rescaling each layer's update by the size of its own weights so that training stays stable at batch sizes where ordinary methods diverge. Lion Optimizer takes a deliberately spare approach, keeping only the sign of its momentum-derived direction and thereby storing far less state per weight — a meaningful saving when a model has billions of them. Lookahead Optimizer is different in kind: rather than replacing an optimizer it wraps one, letting the inner method take several steps before pulling the weights part of the way towards where it arrived, which damps oscillation.

Learning-Rate Schedules

Whatever optimizer is chosen, the overall step size rarely works best held constant. Early in training the weights are far from anything useful and can afford bold moves; later, near a good solution, large steps only bounce around it. A schedule varies that scale over the course of a run.

Learning Rate Warmup covers the opposite risk at the very beginning: with freshly initialised weights and adaptive statistics not yet meaningful, a full-size step can do real damage, so the rate is raised gradually from near zero over the first stretch of training. It is close to mandatory for large models. Cosine Annealing Schedule then decays the rate along a smooth curve that falls slowly at first and steeply near the end, which has become the common partner to warmup.

OneCycle Learning Rate packages both movements into a single deliberate arc — up to a peak, then down well below the starting value — and often reaches a good solution in noticeably fewer epochs. Reduce LR on Plateau takes a reactive approach instead of a planned one: it watches a validation measure and cuts the rate whenever improvement stops, which requires no advance guess about how long training will take.

Gradient Handling & Precision

The last group is about keeping long runs alive on real hardware. Gradient Clipping guards against the exploding case by capping the size of the update before it is applied — if the gradient exceeds a threshold it is scaled back to it, preserving direction while limiting distance. One anomalous batch can otherwise undo hours of work, and clipping is standard wherever gradients are known to spike.

Gradient Accumulation solves a memory problem: when a desirable batch will not fit in memory, several smaller batches are run in turn and their gradients summed before a single update is applied. The effect approximates the larger batch without ever holding it at once, which is how modest hardware trains models designed for far more.

Mixed Precision Training speeds matters up by storing and multiplying most values in a lower-precision number format while keeping a full-precision copy of the weights for the actual update. It roughly halves memory use and runs substantially faster on hardware built for it. The catch is that small gradient values can fall below what the reduced format can represent and vanish to zero, so Loss Scaling multiplies the loss by a large factor before the backward pass — lifting those small numbers into representable range — and divides the result back out before the weights are updated.

Regularization & Normalization

A network with enough capacity can memorise its training data outright, reproducing it faithfully while failing on anything new. Separately, a deep stack can train badly simply because the scale of the values flowing through it drifts from layer to layer. The techniques here address those two problems — one about generalisation, one about stability — and they are grouped together because in practice almost every modern model uses several of them at once.

Regularization & Sparsity

The best-known method deliberately damages the network during training. Dropout switches off a random subset of units on every step, so no unit can rely on any particular neighbour being present and the network is forced to spread its representation across many paths rather than concentrating it in a few. At test time everything is switched back on. DropConnect applies the same intuition one level lower, dropping individual connections rather than whole units. Stochastic Depth goes one level higher again and skips entire layers at random during training, which both regularises and shortens the average path through a very deep stack.

A second approach constrains the weights themselves rather than the activations. Weight Normalization separates each weight vector into a direction and a magnitude and lets the two be learned independently, which makes optimisation better behaved. Spectral Normalization limits how much a layer can stretch any input passing through it, capping its amplification; this is the standard stabiliser where one network is trained against another and unchecked amplification would let training diverge.

A third target is the labels. Label Smoothing stops asking the model for absolute certainty, replacing a target of complete confidence in one category with something slightly softer spread across the others. A model trained this way is less prone to extreme over-confidence and usually calibrated better, at almost no cost.

Normalization Layers

The layers in this group solve the stability problem directly: they re-centre and re-scale the values passing through the network so that each layer receives inputs in a predictable range, whatever the layers below have done. The differences between them come down to a single question — which values are grouped together when computing the statistics.

Batch Normalization computes them across the examples in the current mini-batch. It was the breakthrough that made many deep networks trainable and it remains dominant in vision, but its dependence on the batch is also its weakness: with very small batches the statistics grow unreliable, and training and inference must behave differently, which adds complexity.

Layer Normalization avoids that entirely by computing statistics across the features of each individual example, so behaviour does not depend on batch size or on what other examples happen to be present. That independence is why it, rather than the batch variant, is standard in transformers and in sequence models generally. Instance Normalization narrows the grouping further, normalising each channel of each example separately, which suits image-styling work where per-image contrast should be neutralised. Group Normalization sits between the last two, splitting channels into groups and normalising within each — giving batch-independence while retaining some of the sharing that pure per-channel treatment gives up.

Data Augmentation

The final approach adds no term to the model at all and instead enlarges what it learns from. Data Augmentation transforms training examples in ways that leave their meaning intact — flipping, cropping, shifting colour — so the model sees many variants of each and learns which differences are irrelevant. It is often the single most effective remedy for overfitting when data is limited, because it attacks the shortage at its source.

Two methods extend the idea past single-example transformations by combining examples. Mixup blends two training images and their labels in the same proportion, teaching the model to behave predictably between examples rather than only at them. CutMix instead cuts a patch from one image, pastes it into another, and mixes the labels according to the area each occupies, which keeps local detail sharp where blending would blur it.

Convolutional Networks

A fully connected layer treats every input value as unrelated to every other, which for an image throws away the most useful fact about it: nearby pixels belong together, and a pattern is the same pattern wherever it appears. Convolutional networks build both of those facts into the architecture, and in doing so reduce the number of parameters enormously while improving what the model learns.

Convolution Concepts

The core idea is the Convolution Operation: a small window slides across the input, and at every position the values under it are combined into a single output. Because the same window is used everywhere, a pattern learned in one corner is automatically recognised in every other — the property that makes these models so efficient on images. A layer built from this operation is a Convolutional Layer.

The sliding window itself is the Kernel (Filter), a small grid of learned weights. A kernel is a pattern detector: one may respond to vertical edges, another to a particular texture, and what each detects is learned rather than designed. Sweeping one kernel across the input produces a Feature Map, a grid recording how strongly that pattern was found at each location. A layer applies many kernels at once, and each of the resulting maps is a Channel — the same word describing the three colour channels an image arrives with and the dozens or hundreds of learned maps a layer produces.

Two settings control how the window moves. The Stride is how far it shifts between positions: a stride of one examines every location and preserves size, while a larger stride skips positions and shrinks the output. Padding (Same, Valid) governs the edges, where the window would otherwise hang over the boundary — adding a border of zeros keeps the output the same size as the input, while refusing to pad means only fully-covered positions are computed and the output shrinks slightly at every layer.

Stacking such layers produces an important cumulative effect. Any single output value depends only on a small patch of the layer below, but that patch depends on a larger patch below it, and so on. The Receptive Field is the region of the original input that ultimately influences one value. It starts tiny and grows with depth, which is why early layers detect edges and textures while later ones respond to whole objects — the later ones are simply looking at far more of the picture.

Convolution Variants

The basic operation has been varied in several directions, mostly to buy the same modelling power for less computation. The 1x1 Convolution uses a window covering a single position, which sounds pointless until you notice it still spans every channel: it mixes and re-weights channels without touching spatial layout, and is the standard cheap way to increase or reduce channel count.

Depthwise Convolution takes the opposite decomposition, applying a separate kernel to each channel independently and never mixing them. Neither half is sufficient alone, but the Separable Convolution chains them — a depthwise pass for spatial structure followed by a pointwise pass for channel mixing — achieving close to what a full convolution does at a fraction of the cost. This factorisation is the basis of most architectures designed for phones and embedded hardware.

Two further variants change the geometry. Dilated (Atrous) Convolution spreads the kernel's sampling points apart, leaving gaps between them, so the receptive field grows quickly with depth without adding parameters or reducing resolution — valuable where fine detail must be preserved. Transposed Convolution runs in the opposite direction, increasing spatial size rather than reducing it, and is how a network that has compressed an image down to a small representation expands it back to full resolution.

Pooling

Alongside convolution sits a second, simpler operation for reducing spatial size. A Pooling Layer summarises each small neighbourhood into one value. It has no learned parameters at all; it simply shrinks the grid, which reduces computation in every layer above and makes the representation less sensitive to small shifts in position.

Max Pooling keeps the largest value in each neighbourhood, preserving the strongest response and discarding the rest — appropriate when what matters is whether a pattern was found rather than exactly where. Average Pooling takes the mean instead, retaining a smoother summary of the whole neighbourhood. Global Average Pooling applies that averaging across an entire feature map at once, collapsing each channel to a single number. This is now the usual way to finish a convolutional network, replacing the large fully connected layers that older designs used and removing most of their parameters along with them.

Recurrent Networks (RNN/LSTM)

The networks earlier in this chapter treat each input independently: an image is classified without reference to the one before it. Sequences are different. The meaning of a word depends on the words preceding it, and the next value in a series depends on its history. Recurrent architectures address this by giving a network a memory that persists as it moves along a sequence.

The Recurrent Neural Network (RNN) is the basic form, and its idea is a loop: the same small network is applied at every position, and alongside each new input it receives its own output from the position before. What travels along that loop is the Hidden State, a vector summarising everything the network has seen so far. Because the same weights are reused at every step, a recurrent network can handle sequences of any length.

That elegance carries a serious defect. Applying the same transformation repeatedly means the signal is multiplied by the same weights again and again, and over a long sequence the gradients either fade to nothing or grow without bound. In practice a simple recurrent network struggles to connect events more than a handful of steps apart, which is precisely what most sequence tasks require.

The Long Short-Term Memory (LSTM) was designed to fix exactly this. Its solution is to add a second pathway, the Cell State, which runs along the sequence with only gentle, mostly additive modification — a conveyor belt that information can ride for many steps without being repeatedly transformed. What is added to or removed from that belt is controlled by small learned gates, each producing values between zero and one that determine how much of a signal passes.

There are three. The Forget Gate decides how much of the existing cell state to retain, allowing the network to discard information that is no longer relevant. The Input Gate decides how much of the newly computed candidate information to write in. The Output Gate decides how much of the cell state to expose as the hidden state at this position. Together they let the network hold something for hundreds of steps and release it exactly when needed, which is what the plain recurrent form could not do.

The Gated Recurrent Unit (GRU) pursues the same goal with fewer parts: it merges the cell state back into the hidden state and uses two gates rather than three. The Update Gate (GRU) combines the roles of forgetting and writing in one decision about how much of the state to replace, while the Reset Gate (GRU) controls how much of the previous state is consulted when forming the new candidate. It trains faster and performs comparably on many tasks, so the choice between the two is usually empirical.

Several structural variations apply to any of these. A Bidirectional RNN runs two recurrences, one forward along the sequence and one backward, and combines them — so each position is informed by what follows as well as what precedes. This is valuable when the whole sequence is available at once, and impossible when output must be produced as input arrives. A Stacked RNN places several recurrent layers on top of one another, each reading the sequence of states produced by the one below, adding depth in the feature direction as well as along time.

Training these models needs two adaptations of its own. Truncated BPTT limits how far back the error is propagated: rather than unrolling a sequence of thousands of steps in full, which is prohibitive in memory and badly behaved numerically, the sequence is processed in segments and gradients are carried back only within each. Teacher Forcing addresses a separate difficulty in models that generate one element at a time. During training the true previous element is fed in rather than the model's own prediction, which makes learning far faster and more stable — though it also means the model never practises recovering from its own mistakes, a gap that shows at generation time.

Transformer (Foundations)

Recurrent models process a sequence one position at a time, which makes them inherently serial and leaves distant positions weakly connected. The architecture in this section discards recurrence entirely. Every position can consult every other directly, in one operation, and the whole sequence is processed in parallel — the change that made training on very large datasets practical and that underlies essentially all current language models.

Attention Mechanics

The mechanism at the centre is an Attention Mechanism: rather than compressing history into a single running state, each position looks over all available positions and draws from them selectively, weighting each by how relevant it is. Nothing needs to survive a long chain of transformations, because everything is one step away.

The selection works through three vectors derived from each position. The Query Vector represents what this position is looking for. The Key Vector represents what a position offers as a means of being found. The Value Vector carries the content actually retrieved once a match is made. Comparing one query against one key yields an Attention Score, a number saying how much this position should attend to that one; the scores across all positions are turned into weights, and the values are blended in those proportions.

The standard form of this computation is Scaled Dot-Product Attention. The comparison is a dot product, which is cheap and parallelises well, and the result is divided by a factor related to the vector size — without that division the scores grow large as dimensions increase, the weighting collapses onto a single position, and gradients through it nearly vanish. The scaling is a small detail with outsized importance.

When queries, keys and values all come from the same sequence, the operation is Self-Attention Mechanism, and it is how a model builds context-aware representations: each word's representation becomes a blend of the words it depends on. Doing this once forces a single set of relationships to serve every purpose, so Multi-Head Attention runs several attention operations in parallel over different learned projections and concatenates the results — one head may track grammatical agreement while another follows subject matter. Cross-Attention instead draws queries from one sequence and keys and values from another, which is how a model conditions its output on a separate input such as a source-language sentence.

Two forms of masking constrain what may be attended to. Causal (Masked) Attention prevents a position from seeing anything that comes after it, which is essential when a model is trained to predict the next element — without it the answer would be visible in the input. Padding Mask serves a mundane but necessary purpose: sequences of different lengths are padded to a common size so they can be batched, and this mask stops attention from treating that filler as real content.

Architecture Blocks

The Transformer assembles those mechanics into a repeating block, stacked many times over. Two variants of the block exist. The Encoder Block reads an entire input at once with unrestricted self-attention, producing a representation of it — the right choice when the whole sequence is available and the task is to understand it. The Decoder Block generates output one position at a time, using causal attention so it cannot look ahead, and typically also cross-attending to an encoder's output when one is present.

Attention alone only mixes information between positions; something must also transform each position's own representation. That is the Position-wise Feed-Forward Network, a small network applied identically and independently at every position, which alternates with attention throughout the stack and accounts for a large share of the model's parameters.

Because attention treats its input as an unordered collection, word order would otherwise be invisible — the same words rearranged would produce identical output. Positional Embeddings supply the missing information, adding a representation of each position to its content so that order becomes part of what the model sees. Finally, Pre-LN vs Post-LN names a placement decision within the block: normalising before each sub-layer rather than after it makes deep stacks markedly easier to train and reduces the need for careful warmup, which is why the earlier arrangement has largely given way to the former.

Generation Modes

Two patterns describe how such a model produces output. An Autoregressive model generates one element at a time, appending each to its input before producing the next, so every element is conditioned on everything already written. This is how text generation works, and it is what causal masking exists to support during training.

Sequence-to-Sequence describes the broader shape of mapping one whole sequence to another of possibly different length — translation being the classic case. It predates this architecture and was originally built with recurrent models, but pairing an encoder for the source with an autoregressive decoder for the target is the form it now usually takes.