Notes
🔍

NAS-Bench

Comparing architecture-search methods used to be close to impossible: every paper ran its own search on its own budget, so a reported win might have come from the algorithm or from the thousand GPU-hours behind it. The NAS-Bench datasets removed the excuse by training the architectures once and publishing the results.

A search space with the answers precomputed

The idea is to fix a small, finite space of architectures and exhaustively train every one of them, recording accuracy at several epochs along with training cost. A search algorithm then runs against a table lookup instead of a GPU: querying an architecture takes microseconds, so a method that would have needed days can be repeated hundreds of times with different seeds. The family has grown in useful directions.

  • NAS-Bench-101 — roughly 423,000 unique cell architectures, each trained on CIFAR-10.
  • NAS-Bench-201 — a smaller cell space but results on three datasets, with full training curves, so multi-fidelity methods can be tested too.
  • NAS-Bench-301 — a surrogate benchmark: the DARTS space is far too large to enumerate, so a model trained on a sample predicts the rest.

Why it matters

These benchmarks turned architecture search into something you can study on a laptop, and the findings were deflating in a productive way: with a sensible budget, random search proved competitive with several published methods, which pushed the field into reporting variance and honest baselines. The caveat is that a fixed benchmark space is not the space you care about, so a strong NAS-Bench result is evidence about a method, not a guarantee about your problem.

NAS-Bench is a family of lookup-table benchmarks for neural architecture search. Every architecture in a small, fully enumerated search space is trained in advance and its accuracy and cost recorded, so a search algorithm queries a stored result instead of training a network. NAS-Bench-101 covers 423,000 CIFAR-10 cells; NAS-Bench-201 adds three datasets and full training curves. They make NAS methods reproducible and fairly comparable on a laptop.

Imagine testing every recipe in a cookbook, writing down exactly how each one tasted, and publishing the whole table. Anyone who wants to invent a smarter way of finding good recipes can now look up answers instead of cooking for a month. NAS-Bench is that table for neural network designs. Researchers trained every design in a small, carefully defined set — hundreds of thousands of them — and recorded how well each performed. A new search method can now be tested in minutes on an ordinary computer, and two methods can be compared fairly, because they are answering exactly the same questions.