Automated Model Selection

Ask three experienced practitioners which model to use on an unfamiliar table and you get three answers, all defensible and none verified. The honest position is that nobody knows until the candidates have been tried on this particular data. Automated model selection turns that admission into a procedure.

How it works

The system fixes a pool of learners — regularised linear models, gradient-boosted trees, random forests, nearest neighbours, small neural networks — and compares them under one protocol: the same resampling split, the same metric, the same budget. Because a learner's ranking shifts with its settings, selection and tuning are solved together as the CASH problem rather than one after the other. Three devices keep the bill down:

Multi-fidelity evaluation — score candidates on a subsample or a few epochs and promote only the survivors, as successive halving does.
Meta-learning — begin with the learners that won on datasets with similar meta-features.
Portfolios — a small complementary set of configurations known to cover a wide range of datasets between them.

Why it matters

Default-model habits are expensive and invisible. A boosted tree is the right first guess on tabular data so reliably that the habit goes unexamined, yet on your data it can lose to a linear model that trains in a second and ships in a spreadsheet. The discipline the automation brings is the comparison protocol: FLAML and auto-sklearn rank every candidate on identical folds, and because the winner was chosen on those folds, its score is optimistic — the honest number comes from a test set the selection never touched.

Automated model selection is the automatic choice of which learning algorithm to use for a dataset, decided by scoring candidates under one fixed validation protocol instead of by habit or intuition. Because an algorithm's ranking depends on its settings, selection is solved jointly with tuning as the CASH problem, accelerated by cheap low-fidelity evaluations, meta-learning warm starts and precomputed portfolios. It matters because the best algorithm for a dataset cannot be predicted in advance.

There is no single best way to teach a computer from data, any more than there is a single best tool in a toolbox. A method that works beautifully on medical measurements can be hopeless on shop receipts, and nobody can tell reliably in advance which it will be.

Automated model selection settles that argument by experiment instead of opinion. The software tries a whole shelf of methods on your data, judges every one of them the same way — on examples it was not allowed to see while learning — and keeps the winner. To stay affordable it gives the weak candidates only a quick look and saves the real computing time for the promising ones.