One-Shot NAS

Architecture search spent its first years paying the same bill over and over: propose a network, train it from scratch, read its score, throw the weights away. One-shot NAS refuses to pay twice. Every candidate architecture is treated as a sub-graph of one big network, and that single network is trained once.

The supernet and weight sharing
The oversized parent is the supernet: it contains every operation the search space allows, and any legal architecture is a path through it. Because an operation's weights are shared by every architecture that uses it, training the supernet trains all candidates at once. Evaluating a candidate then costs one forward pass over validation data with its inherited weights — seconds, not GPU-days. Recipes differ in how paths are sampled during training: path dropout zeroes random operations, while single-path methods train exactly one uniformly sampled architecture per step.

Where it is weakest
Inherited weights are a proxy, and the correlation between a sub-network's one-shot score and its accuracy after honest retraining is frequently weak. Operations co-adapt inside the supernet, high-capacity paths get optimised at the expense of others, and candidates that share parameters interfere with each other. A one-shot search can rank architectures confidently and still crown a mediocre one, which is why finalists are always retrained standalone.

The family it spawned
Nearly every affordable NAS method is one-shot in some form: ENAS puts an RL controller over shared weights, DARTS makes path selection differentiable, single-path uniform sampling trains the supernet then searches with evolution, and Once-for-All turns the supernet itself into the deliverable.