Notes
🔍

RL-Based NAS

Designing a neural network is a sequence of discrete choices — this many filters, that kernel size, a skip connection here — and discrete choices cannot be differentiated. RL-based NAS takes the obvious reframing: treat architecture design as decision-making and let a reinforcement-learning agent make the decisions.

How the loop runs
A controller, usually a recurrent network, emits an architecture one token at a time — layer type, width, kernel size, connectivity. That child network is then built, trained, and scored on a validation set, and its accuracy is the reward. Since the reward arrives only after a full training run, the controller is updated by policy gradient, with REINFORCE or PPO.

  • Zoph and Le's original formulation reached competitive CIFAR-10 results at a cost of thousands of GPU-days.
  • NASNet cut that by searching for a reusable cell instead of a whole network, then stacking it.
  • ENAS cut it further with weight sharing, training one supernet so sampled children need no training from scratch.
  • MnasNet folded on-device latency into the reward, making the search multi-objective.

Why it matters in AutoML
RL-based search is the historical anchor of NAS and remains the natural fit when the objective is non-differentiable — accuracy, latency, energy, or any mixture measured on real hardware. Its weakness is sample efficiency: every reward costs a training run, which is exactly what gradient-based and one-shot methods were invented to avoid.

RL-based NAS treats neural-architecture design as a reinforcement-learning problem: a controller network emits an architecture as a sequence of discrete choices, the resulting child network is trained, and its validation accuracy becomes the reward that updates the controller by policy gradient. It was the first approach to make neural architecture search work at scale and still suits non-differentiable objectives such as on-device latency, but each reward costs a full training run.

Imagine an architect who cannot calculate whether a design will stand, so instead builds each one and sees what happens — then slowly learns which ideas tend to work. That is RL-based NAS. One program proposes a design for a neural network: how many layers, how wide, what connects to what. The design is built and trained, and its test score comes back as a reward. Good scores make similar proposals more likely next time. It works, and it discovered several famous networks, but it is expensive: every guess has to be trained before anyone knows whether it was any good.