Notes
🔍

Black-Box Optimization

Some functions refuse to explain themselves. You hand in a set of inputs, wait, and receive one number back — no formula, no gradient, no hint about which direction to move next. Black-box optimization is the craft of making good decisions under exactly that handicap.

What makes a problem black-box

The optimiser is allowed to query the objective and nothing else. Validation accuracy as a function of a model's hyperparameters is the textbook case in AutoML: it has no analytic form, each query costs a full training run, and the answer comes back noisy. Gradient-based methods are unusable here, so all the intelligence goes into choosing where to sample next:

  • Bayesian optimisation fits a cheap surrogate of the objective — a Gaussian process, or the random forest that SMAC uses — and picks the next point by maximising an acquisition function such as expected improvement.
  • Evolutionary methods such as CMA-ES keep a population of configurations and mutate the good ones.
  • Random search remains a serious baseline: it needs no surrogate and parallelises perfectly.

Why it matters in AutoML

The black-box framing is what lets a single optimiser drive wildly different jobs — hyperparameter tuning in Optuna or Hyperopt, pipeline configuration, neural architecture search — because each of them reduces to "propose a configuration, get a score". The price is sample efficiency: with a budget of tens or hundreds of evaluations, every wasted query hurts, which is why multi-fidelity schedulers like Hyperband deliberately peek inside the box and judge candidates from short, cheap partial training runs.

Black-box optimization is the search for inputs that maximise or minimise a function you can only evaluate, never inspect: no formula, no gradient, and each query is expensive. AutoML is framed this way because scoring one configuration means training a model. The framing matters because it dictates the method — sample-efficient strategies such as Bayesian optimisation, evolutionary search and bandit-style budget allocation, rather than gradient descent.

Picture a machine covered in unlabelled dials. You can set the dials, press a button, and see how good the result is — but you cannot open the machine to find out why. Every press costs you a coin.

That is black-box optimization: hunting for the best settings when all you can do is try something and measure the outcome. It describes AutoML almost perfectly, because testing one set of settings means training a whole model, which takes real time and real money. So the clever part is never trying everything — it is choosing which few attempts are worth the coins, and learning from each result what to try next.