Notes
🔍

ENAS

The architecture search behind NASNet was both a landmark and a warning: it found an excellent network and spent on the order of 2,000 GPU-days doing it, because every candidate was trained from scratch and then discarded. ENAS — Efficient Neural Architecture Search — attacked the discarding rather than the searching.

One graph, shared weights

ENAS treats the whole search space as a single large computational graph and every candidate architecture as a subgraph of it. All subgraphs draw on the same pool of parameters, so training one child model leaves its learned weights behind for every other architecture that reuses those operations. The search runs as two interleaved loops:

  • a controller LSTM samples a subgraph — which operation sits at each node, and which earlier nodes feed it;
  • the sampled child trains briefly on the shared weights, and its validation accuracy becomes a reward that updates the controller with REINFORCE.

Because no child is ever trained from zero, the full CIFAR-10 search finishes in under sixteen hours on one GPU — roughly a thousandfold reduction against the reinforcement-learning search it descends from.

Its real legacy

Weight sharing is the idea that outlived the algorithm: one-shot and differentiable methods such as DARTS stand on the same supergraph. The caveat is just as durable. Accuracy measured under shared weights ranks architectures only loosely compared with accuracy after full retraining, and controlled studies of the search phase found ENAS-style search barely ahead of random sampling within its own space. Treat it as a cheap prior over architectures, not as an oracle.

ENAS (Efficient Neural Architecture Search) is a reinforcement-learning architecture search in which every candidate is a subgraph of one shared computational graph, so child models inherit weights instead of training from scratch. An LSTM controller samples subgraphs and is updated by REINFORCE on their validation accuracy. It cut search cost from thousands of GPU-days to under a day on a single GPU, and its weight-sharing trick underpins every later one-shot and differentiable NAS method.

Think of auditioning a band. The slow way is to hire each lineup, rehearse for a month, then start again from nothing with the next one. The fast way is to keep a single rehearsal room where everybody already knows the songs, and simply try different combinations of players.

ENAS does the second thing for neural networks. Rather than building and training every candidate design from zero — which used to take months of computer time — it keeps one big shared network and samples small designs out of it, reusing everything already learned. A separate piece of software watches which combinations score well and proposes better ones. The search drops from months to about a day.