Algorithm Selection (AutoML)
There is no single learning algorithm that wins on every dataset. A gradient-boosted tree that dominates a tabular credit-scoring problem can lose to plain logistic regression on a small, noisy one. Algorithm selection is the part of an AutoML system whose entire job is to answer the question a practitioner normally answers out of habit: which learner should we even try here?
How it works
Algorithm selection treats the learner as something to be searched for rather than assumed. It is rarely separated from tuning, because an algorithm is only as good as its hyperparameters — so AutoML systems attack both at once as the CASH problem (Combined Algorithm Selection and Hyperparameter optimisation), where one search space holds the choice of learner and, nested under each choice, that learner's own parameters. Two approaches dominate:
- Search-based — evaluate candidate learners against a shared time budget and let the optimiser concentrate on the ones that score well, as Auto-sklearn and FLAML do.
- Meta-learning-based — summarise the dataset with meta-features (row count, dimensionality, class imbalance, feature types), then recommend the learners that worked on similar datasets before. Auto-sklearn uses this to warm-start its search instead of beginning cold.
Why it matters
A search that never considers the right model family cannot be rescued by tuning, so this decision sets the ceiling on everything downstream. It also controls cost: a well-chosen algorithm portfolio of three or four complementary learners reaches a strong result in a fraction of the compute a brute-force sweep over every estimator in the library would burn.
Algorithm selection is the problem of deciding which learning algorithm to apply to a given dataset instead of defaulting to a favourite. No method performs best everywhere, so AutoML systems treat the choice as a search problem — usually solved jointly with hyperparameter tuning, a formulation called CASH — guided by cross-validated scores or by meta-features that summarise the data. It matters because no amount of tuning rescues a model family that fits the data badly.
Suppose you need to get across a city. Bike, bus, car or train — the best choice depends on the distance, the traffic and what you're carrying, and someone who always drives will sometimes be stuck.
Machine learning has the same problem. There are dozens of learning methods, and which one works best depends entirely on the data in front of you. Algorithm selection is the job of picking it. Automated systems handle this by quietly trying several candidates on your data, scoring how well each one does, and keeping whichever actually performs best — so you don't need to know the options by name, or have a favourite you reach for out of habit.