Notes
🔍

Network Width

Think of a network layer as a team of parallel feature detectors. Its width is the number of units—or, in other architectures, channels or hidden features—working side by side in that layer. A wider network gives the model more simultaneous “places” to store and transform useful patterns.

What width changes
In a fully connected layer, width is the number of neurons; in a convolutional layer, it is usually the number of output channels; in a transformer, it appears as the hidden dimension, such as d_model or the feed-forward expansion size. If a layer maps 512 input features to 2,048 hidden features, it is wider than one mapping 512 to 512. That extra width increases the number of learnable weights and the variety of intermediate representations the layer can form.

Capacity, not a free upgrade
A sufficiently wide feed-forward network can, in principle, approximate a very broad class of functions. But width is not merely a theoretical dial: it changes training cost and behavior.

  • More width raises representational capacity and can make optimization easier by providing many alternative paths through parameter space.
  • More parameters consume more GPU memory, increase matrix-multiplication cost, and can require more data or stronger regularization.
  • Too little width creates a bottleneck: the network compresses information too aggressively and can underfit even when trained well.
  • Excessive width can fit training data extremely well while generalizing poorly, particularly on small datasets.

How it appears in practice
In a ResNet, widening the channel count gives each block more feature maps; in a transformer, increasing the hidden size gives every token representation more coordinates. A model whose training loss remains high after sensible optimization may be too narrow, while a model with a widening gap between training and validation loss needs regularization such as weight decay, data augmentation, or dropout. Width therefore controls how much a network can represent at each stage, while depth controls how many successive transformations it can compose.

Network width is the number of neurons, channels, or hidden features in a layer. Wider layers can represent more features in parallel and generally increase a network’s expressive capacity and parameter count. Width matters because insufficient width limits the functions a network can learn, while excessive width raises memory and computation costs and can require stronger regularisation.

Imagine a team solving a puzzle. A wider network has more people working side by side at each stage, so it can notice more different clues at once. A narrower one has fewer people, so it may miss some patterns or need more stages to handle the same job.

In an AI neural network, network width means how many small decision-making units are placed in a layer. More width gives the network more room to recognize complicated details: many features in a photo, subtle tones in language, or unusual patterns in data. But a wider network also takes more computing power and can become overly focused on its training examples.