Weight Sharing (NAS)

The first neural architecture searches were absurdly expensive because every candidate network was trained from scratch, thousands of times over. Weight sharing is the trick that broke that cost: train one over-parameterised network, and let each architecture you want to evaluate borrow its weights instead of learning its own.

One supernet, many subgraphs
The search space is encoded as a single large network — a supernet, or one-shot model — in which every candidate architecture appears as a subgraph:

• training samples subgraphs and updates only the weights they touch, so each operation is trained across many architectural contexts;
• evaluating a candidate means activating its subgraph and measuring it, with no training of its own;
• ENAS introduced the idea with a controller sampling subgraphs, DARTS made the choice continuous and differentiable over the same shared weights, and Once-for-All trains a supernet whose subnetworks are directly deployable at different latency targets.

The cost it removes
Searches measured in thousands of GPU-days fell to single-digit GPU-days. That collapse is why architecture search became usable outside large industrial labs at all, and why nearly every modern method is built on some form of sharing.

The bias it introduces
Shared weights rank architectures imperfectly. Operations co-adapt to the supernet rather than to the standalone network they will become, and benchmarks such as NAS-Bench-201 show the correlation between one-shot score and honestly retrained accuracy can be weak. Two consequences follow: the winner is always retrained from scratch before anyone believes it, and a search can confidently return a cell that disappoints.