NASNet (Algorithm)
Searching for an entire neural network layer by layer is hopeless — the space of possible wirings is astronomically large. NASNet got around that by asking a much smaller question: find one good building block, then stack copies of it until the network is as big as the task needs.
A controller that writes architectures
A recurrent network, the controller, emits a sequence of choices describing a cell: which two earlier feature maps to combine, which operation to apply to each (separable convolutions of several sizes, pooling, identity), and how to merge the results. Two cell types are searched together — a normal cell that preserves spatial size and a reduction cell that halves it. Each proposed cell is assembled into a small network, trained on CIFAR-10, and its validation accuracy becomes the reward in a REINFORCE policy-gradient update, so the controller shifts probability toward cells that score well.
Search small, deploy large
The payoff is transfer. Cells discovered on CIFAR-10 were re-stacked deeper and wider to build NASNet-A for ImageNet, where it outperformed the hand-designed architectures of its day at comparable cost. The search itself consumed roughly 2,000 GPU-days — the price of training thousands of candidates from scratch.
Why it still matters
NASNet fixed the vocabulary that later work inherited: the cell-based search spaces DARTS, ENAS and AmoebaNet explore are all descendants of it. It also made the compute bill impossible to ignore, which is precisely what motivated weight sharing and one-shot search. ScheduledDropPath, the regulariser introduced alongside it, is still standard when training stacked-cell networks.
NASNet is the reinforcement-learning architecture search that made cell-based design practical. A recurrent controller proposes a small normal cell and reduction cell, each candidate is trained on CIFAR-10, and validation accuracy serves as the policy-gradient reward. The discovered cells are then stacked deeper and wider to scale up to ImageNet. NASNet set the cell-based search space that DARTS, ENAS and AmoebaNet reuse, and exposed the enormous compute cost of training every candidate.
Think about designing a tall building. Rather than drawing every floor separately, an architect designs one good floor plan and repeats it all the way up. NASNet does that for neural networks. A small controller program suggests a single building block, the block is stacked into a network, the network is tested, and the score tells the controller whether to suggest something similar next time. After thousands of attempts it lands on a block that works beautifully — and the same block can then be stacked higher for harder jobs. It was one of the first times a computer designed a network that beat the ones human experts had drawn by hand.