Notes
🔍

Evolutionary NAS

Evolution designed the eye without calculus, by keeping what worked and varying it. Evolutionary NAS applies the same bargain to neural networks: hold a population of architectures, breed the good ones, drop the rest, and let the design improve across generations without anyone writing down what a good design looks like.

How a generation works

A run maintains a population of candidate architectures, each trained briefly and scored on validation data. The loop is deliberately simple:

  • Selection — sample a handful of individuals at random and take the best, a tournament;
  • Mutation — copy the winner and change one thing: swap an operation, rewire an edge, alter a filter count;
  • Evaluation — train the child briefly and add it to the population;
  • Removal — retire one individual so the population size stays fixed.

What gets retired turns out to matter. Regularized evolution, also called aging evolution, removes the oldest individual rather than the worst, which stops the population from locking onto one lucky early result and forces good architectures to keep re-proving themselves. That variant produced AmoebaNet, the first evolved architecture to match and then beat hand-designed and RL-searched networks on ImageNet.

Why pick evolution

It needs no gradients and no differentiable relaxation, so the objective can be anything you can measure — latency on a phone, energy per inference, parameter count — and the search space can mix discrete, conditional and structural choices freely. That makes it the natural engine for multi-objective search such as NSGA-Net, whose output is a Pareto front of accuracy-against-cost trade-offs rather than one winner. The price is sample efficiency, so it is normally paired with weight sharing, proxy tasks or short training budgets.

Evolutionary NAS searches for neural architectures with population-based optimisation: candidates are scored on validation data, winners are picked by tournament selection, mutated — an operation swapped, an edge rewired — and the population is refreshed each generation. Regularized evolution, which retires the oldest individual rather than the worst, produced AmoebaNet. Because it needs no gradients, it handles non-differentiable goals such as latency, which makes it the standard engine for multi-objective architecture search.

Dog breeds didn't come from a blueprint. Breeders kept the animals with the traits they wanted, bred those, and repeated — and after enough generations you have a sheepdog.

Evolutionary NAS breeds neural networks in the same way. The computer starts with a crowd of network designs, tests each one briefly, keeps the better performers, and makes slightly altered copies of them: one layer changed here, one connection moved there. Weak designs are dropped. Repeat a few thousand times and useful structures appear that no engineer thought to propose. It is slow and hungry for computing power, but it can chase goals that are awkward to write down as equations, such as running fast on a phone.