Notes
🔍

AutoML-Zero

Most architecture search starts from a generous head start: convolutions, attention blocks, backpropagation and a known loss function are all handed over, and the search merely arranges them. AutoML-Zero asks what happens when you take that scaffolding away.

The setup

Published by Google Brain in 2020, AutoML-Zero evolves entire machine-learning algorithms as short programs over a tiny instruction set — add, multiply, matrix product, random draw — organised into three functions: Setup, Predict and Learn. Nothing about learning is built in. An evolutionary search mutates the programs, scores each on small tasks, and keeps the better ones.

What it rediscovered

  • Linear regression trained by gradient descent, emerging from programs that initially output noise.
  • Two-layer neural networks with backpropagation, assembled instruction by instruction.
  • Techniques resembling weight averaging, gradient normalisation and learning-rate decay — none of them supplied in advance.

Why it matters

AutoML-Zero is not a tool you would reach for on a real dataset; the search is enormously expensive and the tasks are toy-sized. Its value is as evidence about the ceiling of automation. Conventional architecture search inherits human priors and can only recombine them, while a search over raw operations shows the building blocks themselves are discoverable. It marks the far end of the AutoML spectrum: maximum search cost, minimum assumption.

AutoML-Zero is a 2020 Google Brain method that evolves machine-learning algorithms from scratch, using only primitive mathematical operations instead of pre-built layers, losses or optimisers. Candidate programs define setup, prediction and learning steps, and an evolutionary search mutates and selects them. It matters as a limit case: the runs rediscovered gradient descent and backpropagation unaided, showing that human-designed components are themselves searchable rather than fixed.

Imagine handing someone only basic arithmetic — add, multiply, pick a random number — and asking them to invent the idea of learning from experience. No textbook, no hints, nothing borrowed.

That is the experiment behind AutoML-Zero. Researchers let a computer write millions of tiny programs out of those raw operations, kept the ones that did slightly better at a task, and mutated them again and again, rather like breeding. Over time the programs reinvented techniques that human researchers took decades to find. It is far too slow to use for real work. The point was to show that even the ingredients of AI can be discovered, not just assembled by hand.