Differentiable NAS
Searching for a neural-network architecture normally means training thousands of candidates and comparing their scores, which is why the first wave of architecture search ran on hundreds of GPUs for weeks. Differentiable NAS asks a different question: what if the architecture itself were something gradient descent could optimise?
Relaxing a discrete choice
Choosing between a 3x3 convolution, a 5x5 convolution and a pooling layer is a discrete decision, and discrete decisions have no gradient. Differentiable NAS makes the choice continuous. Every candidate operation is kept inside one over-parameterised supernet, each carrying a learned architecture weight, and an edge's output becomes a softmax-weighted mixture of all its operations. Training then alternates two gradient steps:
- ordinary network weights updated on the training split;
- architecture weights updated on a held-out validation split.
When the search ends, the highest-weighted operation on each edge is kept and the resulting network is retrained from scratch. DARTS is the canonical implementation, and it brought searches costing thousands of GPU-days down to roughly one GPU-day.
Where it bites
The supernet holds every candidate operation in memory at once, so the search space stays small — usually a repeated cell rather than a whole network. Searches also collapse: parameter-free skip connections improve fastest early on, accumulate weight, and leave you with a shallow, degenerate cell. PC-DARTS samples a fraction of channels to cut the memory cost, and progressive or early-stopped variants curb the collapse, but the gap between the mixed supernet and the discretised child it hands you remains the method's honest weak point.
Differentiable NAS makes neural architecture search continuous: all candidate operations are combined into a supernet with learned mixing weights, so the architecture is optimised by gradient descent alongside the network weights and then discretised to the strongest operation per edge. Introduced by DARTS, it cut search cost from thousands of GPU-days to about one, which is what made architecture search affordable outside large industrial labs. Its weakness is instability — searches can collapse toward skip connections.
Imagine choosing a route across a city by driving every possible road at once, a little, and gradually pressing harder on the ones that feel fastest. At the end, you keep only the roads you were leaning on.
That is Differentiable NAS. Designing a neural network means picking from a huge menu of building blocks, and testing every combination separately is enormously expensive. So this method blends all the options into one big flexible network, lets ordinary training nudge the blend toward whatever works, and then keeps the winners. It turned network design from a job needing a warehouse full of computers into something a researcher can run overnight on a single one.