DARTS
Early neural architecture search worked by trial and error on an industrial scale: generate an architecture, train it, score it, and repeat for thousands of GPU-days. DARTS — Differentiable Architecture Search — replaced that with something a gradient can walk down, bringing the cost of designing a network down to a few GPU-days.
The trick: relax the choice
Instead of picking one operation for each connection inside a network cell, DARTS places every candidate operation — convolutions of several kinds, pooling, a skip connection, or nothing at all — on the edge simultaneously and takes a weighted sum of their outputs. The weights come from a softmax over continuous architecture parameters, written α. Because the mixture is differentiable, the architecture becomes just another set of parameters to optimise.
Training the supernet
DARTS solves a bilevel problem by alternating two gradient steps: network weights w are updated on the training split, and the architecture weights α on a validation split, so architectural choices are judged on data the weights did not fit. One oversized supernet is trained this way; when it finishes, the strongest operation on each edge is kept by argmax, giving a discrete cell that is then retrained from scratch at full size.
Why it matters, and where it bites
DARTS made architecture search reproducible on a single workstation and spawned the gradient-based NAS family. It also has well-documented failure modes: searches collapse towards parameter-free skip connections, the argmax architecture can score worse than the mixture it came from, and results swing across random seeds. PC-DARTS, Fair-DARTS and robust DARTS exist to patch these behaviours.
DARTS (Differentiable Architecture Search) makes neural architecture search solvable by gradient descent. It relaxes the discrete choice of operation on each edge of a network cell into a softmax-weighted mixture of all candidates, then alternately trains network weights on the training split and architecture weights on a validation split. The final cell is read off by keeping the strongest operation per edge. It cut search cost from thousands of GPU-days to a few, at the price of instability.
Designing the structure of a neural network — how many layers, what kind, wired together how — used to mean building thousands of candidates and training each one, which is enormously expensive. DARTS found a shortcut. Instead of choosing one option at each junction, it tries all of them at once and gives each a confidence score, then nudges those scores up and down as it learns, the way you might adjust several speakers until the mix sounds right. At the end it keeps the loudest option at every junction. That turned weeks of brute force into roughly a day of work.