Neural Architecture Search (NAS)
Picking a model's hyperparameters is one problem; picking its shape is a bigger one. How deep the network goes, how wide each stage is, which operations sit where and what connects to what — those calls have traditionally come from expert intuition and a long series of experiments. Neural architecture search turns them into an optimisation problem and hands them to a machine.
Three components, in every method
Whatever the paper, a NAS system is a choice of three things:
- a search space — the architectures that can be described at all, usually built from repeating cells rather than free-form graphs;
- a search strategy — how candidates are proposed: reinforcement learning, evolutionary mutation, Bayesian optimisation, or gradient descent over a relaxed space;
- a performance estimation strategy — how a candidate is scored without full training: truncated runs, smaller images, weights inherited from a supernet, or a learned accuracy predictor.
Estimation is where the money goes
The first searches trained every candidate from scratch and cost thousands of GPU-days for a single result. Everything since attacks that number — weight sharing in ENAS, differentiable relaxation in DARTS, lookup benchmarks like NAS-Bench-201 for research. The tradeoff is fidelity: cheap estimates rank architectures imperfectly, so a search can confidently return a cell that fails to hold up when retrained honestly.
What it delivers
EfficientNet, MobileNetV3 and most on-device vision backbones came out of searches rather than sketches. Multi-objective variants optimise latency, memory or energy alongside accuracy, which is why NAS earns its cost most clearly when the deployment target is fixed and tight.
Neural architecture search automates the design of a network's structure — depth, width, operations and connections — instead of leaving it to expert intuition. Every method combines three parts: a search space of describable architectures, a search strategy that proposes candidates (reinforcement learning, evolution, Bayesian optimisation or gradient-based relaxation), and a performance estimation shortcut that scores a candidate without training it fully. It produced EfficientNet and MobileNetV3, and it is the costliest branch of AutoML.
Suppose you are building a bicycle, but nobody has told you how many gears it should have, what size the wheels are, or how the frame should be laid out. You could trust a designer's instinct, or you could build thousands of bicycles and keep the best one. Neural architecture search is the second approach, applied to AI models. The computer proposes a layout for a model, trains it, scores it, and uses that score to propose something better — over and over. The payoff is real: many of the AI models running on your phone right now were designed this way, and they are smaller and faster than anything drawn by hand. The catch is the electricity bill.