Notes
🔍

AutoML Benchmark

Every AutoML tool claims to find good models quickly. An AutoML benchmark is the agreed way to check that claim: a fixed list of datasets, a fixed time budget, a fixed metric, and the same evaluation code wrapped around every framework being compared.

What a benchmark fixes

  • The task suite — a curated set of classification and regression datasets, usually drawn from OpenML, chosen to span sizes, class balances and feature types.
  • The budget — wall-clock time and core count, because a framework that wins at ten hours can lose at ten minutes.
  • The protocol — fold splits, the metric (AUC, log loss, RMSE), and what counts as a failure when a run crashes or times out.

The reference implementation

The OpenML AutoML Benchmark is the one most papers cite. It ships containers and thin adapters for auto-sklearn, TPOT, AutoGluon, H2O AutoML and FLAML, so a comparison runs each framework's own defaults rather than someone's hand-tuned settings — the single largest source of unfair results in earlier papers.

Why it matters

Without a shared harness, every framework's own paper reports it as the winner and nothing is learned. A benchmark also exposes behaviour that a single accuracy number hides: how often a tool fails outright, how it degrades on wide sparse data, and whether its leaderboard score survives on data the search never touched. When you are choosing between frameworks, the benchmark table is a far stronger signal than any feature list.

An AutoML benchmark is a standardised evaluation suite for AutoML systems: a fixed collection of datasets, a fixed time and compute budget, a fixed metric, and identical harness code around every framework. The widely used OpenML AutoML Benchmark runs each tool at its own defaults inside containers. It matters because framework-authored comparisons are not reproducible, and only a shared protocol shows which system actually wins at a given budget.

Imagine car magazines testing new models. If every manufacturer ran its own test, on its own track, every car would turn out to be best in class. So reviewers agree on one track, one fuel load and one stopwatch, and drive them all.

An AutoML benchmark is that shared track for model-building software. Researchers pick a set of real datasets, give every tool the same hour and the same computer, and record how good a prediction each one came back with. The result is a comparison table you can actually trust — and it tells you something a tool's own website never will, which is where it quietly fails.