Notes
🔍

Meta-Learning

Train a model on one dataset and you get a model. Train across hundreds of datasets and you can get something else — a sense of what works where. Meta-learning is the deliberate pursuit of that second thing, usually summarised as learning to learn.

The two-level setup

The unit of training stops being an example and becomes a task. Tasks are drawn from a distribution, each split into a small support set for adapting and a query set for scoring. The inner level solves individual tasks; the outer level adjusts whatever the inner level depends on, judged by how well that adaptation went. Three families dominate: optimisation-based methods that learn an initialisation or an update rule, such as MAML and Reptile; metric-based methods that learn a space where nearest-neighbour comparison works from a handful of labels, such as prototypical and matching networks; and model-based methods that keep adaptation inside a network's own memory.

Where it shows up in practice

  • Few-shot classification — recognising a new category from five labelled images.
  • Warm-started search — Auto-sklearn picks starting configurations by matching your dataset against stored ones, which is meta-learning in its most practical form.
  • Learned surrogates — performance predictors trained across datasets, so a tuner can skip configurations that lost elsewhere.

Why it matters

Meta-learning is the research foundation under AutoML's efficiency claims. A search that genuinely starts from scratch spends its budget rediscovering known results; the only way around that is to carry something over from previous problems, and meta-learning is the formal account of what that something is and how to acquire it.

Meta-learning, or learning to learn, trains over a distribution of tasks instead of a single dataset, so the resulting system adapts to a new task from very little data. Approaches learn a fast-adapting initialisation, an update rule, or a comparison space. It matters as the foundation of AutoML's efficiency: transferring experience across datasets is what lets a search begin from informed configurations rather than from nothing.

A child who has met three dogs recognises a fourth breed instantly. They are not working from thousands of photographs — they have learned what kind of thing a dog is, so each new one costs almost nothing to pick up.

Meta-learning is teaching software that habit. Instead of training it on one job until it is expert, you train it across many small jobs so it becomes good at picking up new ones quickly. The payoff is learning from a handful of examples instead of millions, which matters whenever gathering data is slow, expensive, or simply impossible.