Pipeline Search
Tuning a model's hyperparameters is the part of AutoML everyone pictures, but it answers only half the question. How missing values are filled, how categories are encoded, whether features are selected — that frequently matters more than the model's settings. Pipeline search is the broader problem: searching over the structure of the whole pipeline and its hyperparameters together.
What makes it hard
The space is conditional and variable-length rather than a fixed box of numbers. Picking a tree ensemble opens one set of knobs and closes another, and inserting a feature-selection step changes what the next step receives. Each candidate costs a full cross-validated fit. Three strategies dominate:
- Fixed-skeleton Bayesian search — Auto-sklearn fixes the shape (impute, encode, rescale, select, classify) and lets SMAC optimise the choices inside it.
- Genetic programming — TPOT evolves pipelines as trees, so branching and stacked structures emerge instead of being prescribed.
- Grammar-based search — the legal pipelines are generated from a grammar, turning the task into planning over valid sequences.
Why it matters in AutoML
Pipeline search is what lets a tool accept a raw, messy table instead of a clean feature matrix. It is also where the compute goes, which is why every serious implementation pairs it with early stopping, caching of shared pipeline prefixes, and a hard time budget.
Pipeline search is the AutoML problem of choosing the structure of a machine-learning pipeline — which preprocessing, feature-engineering and modelling steps to include, and in what order — jointly with the hyperparameters of each step. It generalises hyperparameter optimisation, because preprocessing choices decide accuracy as much as model settings do. TPOT searches with genetic programming and Auto-sklearn with Bayesian optimisation over a fixed skeleton; both make raw tabular data usable without hand-built preprocessing.
Cooking a dish is not just about the main ingredient. The order of steps matters: chop, marinate, sear, then simmer. Swap those around and the result changes. Building a prediction system works the same way — before the model ever sees your data, it usually has to be cleaned, filled in and reshaped. Pipeline search is the computer trying out whole recipes, not just adjusting the seasoning on one step. It mixes and matches the cleaning steps, the transformations and the model itself, scoring each combination to find one that works. That is what lets you hand over a messy spreadsheet and get something useful back.