Gradient-Based NAS

Searching for a neural network architecture sounds like a problem of trying things: build a network, train it, score it, try another. Gradient-based NAS refuses that framing. It makes the architecture itself differentiable, so the choice of operations can be learned by the same gradient descent that learns the weights.

Relaxing a discrete choice

The move is a continuous relaxation. Instead of committing to one operation on each connection — 3x3 convolution, pooling, skip, none — a supernet holds all of them in parallel and mixes their outputs using a softmax over learnable architecture parameters. Training then alternates two updates: network weights on the training split, architecture parameters on a validation split. When the search ends, the strongest operation on each edge is kept and the pruned network is retrained from scratch. What this buys you:

Cost. One supernet training replaces thousands of individual trainings. DARTS reported a CIFAR-10 search in roughly one GPU-day, against the thousands of GPU-days spent by reinforcement-learning search.
Differentiable side objectives. Latency or parameter count can be folded straight into the loss, which is how hardware-aware variants such as ProxylessNAS and FBNet operate.

The catch

Relaxation changes the problem being solved. The mixed supernet is not the network you ship, so the ranking it produces can mislead: DARTS is known to collapse toward parameter-free skip connections, which train fastest inside the mixture but yield a weak final model. Partial-channel sampling (PC-DARTS), early stopping on the architecture, and regularised variants exist to stabilise it. Memory is the other limit — every candidate operation sits in the graph at once — so search is normally run on one small cell and the winning cell stacked up.

Gradient-based NAS searches neural architectures by making the discrete choice of operations differentiable. All candidate operations are placed in one supernet and mixed by learnable architecture parameters, updated by gradient descent on validation loss alongside the network weights; the strongest operation per connection is then kept and retrained. It matters because it cuts architecture search from thousands of GPU-days to about one, at the cost of a relaxation that can rank candidates unfaithfully.

Picture choosing a route through a city by driving every road, one at a time, and timing yourself. That is slow. Now picture a magic map where you can travel all the roads at once, faintly, see which ones feel most promising, and only then commit to a route.

Gradient-based NAS is that magic map, used for designing artificial brains — the networks behind image and speech recognition. Instead of building and testing thousands of separate designs, the computer builds one blended design containing every option, nudges its preferences a little at a time, and finally keeps the parts that pulled their weight. Work that once needed a room of computers for weeks can finish on a single machine in about a day.