Anytime Performance
Ask an AutoML system for a model and the honest reply is "how long have I got?" Anytime performance is the property of a search that always has a usable answer ready — not only when the budget runs out.
What gets measured
During a run the system holds an incumbent: the best configuration validated so far. Plot the incumbent's score against elapsed time, and that curve — not a single final number — is the real description of the system's behaviour. Two frameworks can finish an hour-long run with identical accuracy and behave nothing alike: one reaches 95% of its final score in the first three minutes, the other spends fifty minutes exploring before anything useful appears. Summary statistics such as the area under the curve, or the time taken to come within 1% of the final score, compress that comparison into something rankable.
Why it matters in AutoML
- Budgets are guesses. A run gets cut short, a deadline moves, a notebook dies — with strong anytime behaviour, whatever you hold when you stop is still worth deploying.
- It drives design. Meta-learned warm starts, portfolios and low-fidelity evaluation schemes such as successive halving exist largely to make the early part of the curve climb faster.
- Benchmarks report it: the AutoML Benchmark evaluates frameworks at several time budgets, because the winner at one hour is sometimes the loser at ten minutes.
Judging a search by its final score alone rewards systems that are only useful to people willing to wait.
Anytime performance describes how good an AutoML system's best-so-far model is at every point during a search, not just at the end. It is read off a curve of the incumbent score against elapsed time or number of evaluations. The property matters because real budgets are uncertain and runs get interrupted: a framework that reaches near-final quality within minutes is more useful than one with the same final score reached only after hours.
Imagine two cooks preparing dinner. Both will serve a fine meal in two hours, but one has something tasty on the plate after fifteen minutes while the other has nothing edible until the very end. If the guests arrive early, only one of them saves the evening.
Anytime performance is that idea applied to automated model building. These systems try out many models and hold on to the best one found so far, and anytime performance asks how good that best-so-far model is at whatever moment you decide to stop. It matters because real deadlines, budgets and patience rarely match the plan, and a system that only pays off at the finish line often never gets there.