Foundations & Concepts

Building a machine-learning model by hand means making a long chain of expert decisions: which algorithm to use, how to prepare the data, and which settings to try. Automated Machine Learning — almost always shortened to AutoML — is the effort to hand those decisions to software, so that a good model can be produced with far less manual tuning and expertise.

The thing being automated is the AutoML Pipeline: the full sequence that turns raw data into a trained, deployable model, spanning data preparation, model choice, and tuning. One of the first decisions inside that pipeline is Algorithm Selection (AutoML) — picking which learning algorithm is likely to work best for the dataset at hand, rather than assuming a favourite in advance.

Every such decision has options, and together they define a Configuration Space — the structured set of all algorithms, preprocessing steps, and hyperparameter values the system may choose from. The broader landscape it explores is often called the Search Space (AutoML). Because the system usually cannot see inside model training to understand exactly why one configuration beats another, it treats the task as Black-Box Optimization: it can try a configuration and observe the score, but must search intelligently rather than reason about the internals.

To search well, AutoML systems lean on Meta-Features (AutoML) — descriptive properties of a dataset, such as its size or class balance, that hint at which approaches have worked on similar data before. All of this ultimately serves accessibility. Low-Code ML tools let people build models with minimal programming, while No-Code ML platforms remove coding entirely, opening model-building to users who are not trained data scientists.

Hyperparameter Optimization Libraries

Every machine-learning model has settings that are chosen before training begins, and the right settings can be the difference between a mediocre model and a strong one. Searching for them by hand is slow, so a family of software libraries has grown up to automate the hunt. This section introduces the tools practitioners actually reach for when they need to tune a model.

The most popular general-purpose tool is Optuna, which lets you describe the search using ordinary Python code and then runs many trials on your behalf, abandoning weak ones early. An earlier and still widely used option is Hyperopt, one of the first Python libraries to make model-based tuning accessible. Both learn from each trial to make the next guess smarter rather than searching blindly.

When tuning outgrows a single machine, Ray Tune steps in: it runs and coordinates many trials across a whole cluster and can drive the other libraries as its search engine. Deciding how to spend limited compute across those trials is the job of clever schedulers. Asynchronous Successive Halving (ASHA) keeps a large cluster busy by promoting promising configurations without waiting for slower ones to finish, while BOHB blends that budget-saving approach with a model that guides which settings to try next.

Some problems call for more specialised optimisers. SMAC is built for messy, expensive search spaces that mix many kinds of settings, and has long powered automated model-building systems. For research-grade work, BoTorch provides low-level building blocks for Bayesian optimisation on top of PyTorch, giving experts full control to design a custom search — for instance, one that balances accuracy against speed.

Neural Architecture Search

Designing a high-performing neural network by hand is slow, expert-driven, and full of trial and error. Neural Architecture Search turns that design process into something a machine can do automatically — searching through possible network structures to discover ones that work better than a human might craft. This section introduces the core search strategies, the landmark algorithms that put them into practice, and the specialised variants that make the search cheaper, broader, or more reusable.

NAS Strategies

At its heart, Neural Architecture Search (NAS) is the task of automatically discovering a good network design instead of hand-engineering it. What separates the approaches is how they explore the vast space of possible architectures, and several distinct strategies have emerged.

The earliest framed the search as a decision process: RL-Based NAS uses reinforcement learning, training a controller that proposes architectures and is rewarded when they perform well. A parallel idea borrows from biology — Evolutionary NAS keeps a population of architectures and repeatedly mutates and selects the fittest. Both are powerful but expensive, because each candidate must normally be trained to be judged.

Later strategies attack that cost. Gradient-Based NAS makes the search itself differentiable, so architecture choices can be optimised with ordinary gradient descent rather than slow trial-and-error. One-Shot NAS trains a single over-arching "supernet" that contains every candidate as a sub-network, letting many designs be evaluated without training each from scratch. And Cell-Based Search shrinks the problem by searching only for a small repeatable building block (a "cell") which is then stacked to form the full network — dramatically reducing what the search must explore.

NAS Algorithms

These strategies came to life through a series of landmark algorithms. NASNet (Algorithm) was an early breakthrough that used reinforcement learning to discover a cell on a small dataset and transfer it to larger ones, while AmoebaNet showed that an evolutionary approach could match or beat it.

The next wave focused on efficiency. ENAS slashed search cost through Weight Sharing (NAS), the key trick of letting candidate architectures reuse a common pool of trained weights instead of each learning its own from zero. DARTS made the search fully differentiable, bringing NAS within reach of a single GPU and becoming one of the most influential methods in the field. Pushing automation to its limit, AutoML-Zero attempts to evolve entire learning algorithms from scratch using only basic mathematical operations, with minimal human priors.

NAS Variants

As NAS matured, specialised variants extended it in useful directions. Differentiable NAS generalises the gradient-based idea into a broad family of methods that treat architecture search as a continuous optimisation problem. Multi-Objective NAS recognises that accuracy is not the only goal, simultaneously optimising for constraints like latency, memory, or energy — essential when targeting phones or embedded devices.

Other variants make the search more practical and reusable. NAS-Bench provides standardised benchmark datasets of pre-evaluated architectures, so researchers can compare methods fairly without enormous compute. A Neural Predictor (NAS) learns to estimate how well an architecture will perform without fully training it, speeding the search further. And Once-for-All Networks take reuse to its conclusion: train one large network a single time, then extract many smaller specialised sub-networks for different devices without any retraining.

Automated Feature Engineering

The columns you feed a model — the "features" — often matter more than the model itself. Crafting good features by hand takes domain knowledge and patience, so a branch of AutoML aims to do it automatically. Automated Feature Engineering is the practice of letting software create, transform, and select input features, turning raw tables into signals a model can learn from.

The starting point is Feature Generation — systematically producing new candidate columns by combining, aggregating, or transforming existing ones. A common first hurdle is non-numeric data, which is why Automated Categorical Encoding handles the conversion of labels and categories into numeric form without manual bookkeeping. A powerful, structured way to generate features from related tables is Deep Feature Synthesis, which stacks simple operations across linked datasets to build rich, meaningful columns.

Several libraries put these ideas into practice. Featuretools is the reference implementation of Deep Feature Synthesis for relational data, while Featurewiz focuses on automatically generating and then selecting the most useful features. OpenFE targets efficient discovery of high-value features for tabular problems, and tsfresh specialises in extracting hundreds of descriptive features from time-series data, where each column is a signal changing over time.

Automated Pipeline Construction

A working machine-learning solution is rarely a single model — it is a chain of steps: clean the data, transform it, select features, train a model, and combine the results. Assembling that chain well is itself a hard problem, and this section is about automating it end to end.

Each step in the chain is a Pipeline Building Block — a reusable component such as a scaler, encoder, or classifier that can be slotted together with others. Stringing these blocks into a complete, valid sequence is ML Pipeline Generation, where the system decides not just which model to use but the whole processing flow around it. Choosing the settings for that flow is Automated Pipeline Configuration, which tunes how each block behaves.

Because the number of possible pipelines is enormous, the system needs a way to explore them. Pipeline Search is that exploration — navigating the space of possible pipelines to find high-performing ones — and it often relies on Configuration Sampling to draw candidate pipelines to try rather than testing every option. Finally, Pipeline Validation (AutoML) checks that a proposed pipeline actually works and generalises, guarding against combinations that look good on paper but fail on new data.

Model Selection & Ensembling

Given a dataset, which model should you trust — and is one model even enough? This section covers how AutoML chooses among candidate models and, often, combines several into something stronger than any single one.

The core task is Automated Model Selection: automatically comparing candidate models and keeping the ones that perform best on the data. A practical way to hedge bets is to keep an Algorithm Portfolio — a curated set of diverse algorithms known to cover different kinds of problems — so the system always has strong options to draw from. Guiding that choice, a Model Recommender suggests promising models based on the dataset's characteristics, while Meta-Model Selection uses knowledge learned from past problems to predict which model will win.

Rather than settle on one winner, AutoML often combines models. Automated Ensembling blends the predictions of several models so their strengths reinforce one another and their individual mistakes cancel out. A particularly effective form is Automated Stacking, where a further model learns how best to combine the others' outputs. The tool that builds such layered combinations automatically is an Auto-Stacker, which assembles and tunes these stacked ensembles without manual design.

AutoML Frameworks (Open Source)

The ideas behind AutoML — automatic model selection, tuning, feature engineering, and ensembling — come together in ready-to-use open-source frameworks. Each packages the whole workflow behind a simple interface, and each has its own philosophy about how much to automate and which problems to prioritise. This section introduces the most widely used ones.

Several grew directly out of research. Auto-sklearn builds on the popular scikit-learn library and adds automatic algorithm selection, tuning, and ensembling. TPOT uses evolutionary search to discover entire pipelines, and AutoKeras brings automation to deep learning, including neural architecture search. AutoGluon is known for producing strong models quickly on tabular, text, and image data with little configuration.

Others focus on speed, breadth, or ease of use. FLAML is designed to find good models fast while using very little compute. H2O AutoML trains and ranks many models and stacks the best into a leaderboard. PyCaret AutoML wraps the workflow in a concise, low-code interface for rapid experimentation, and EvalML pairs automated pipeline building with a strong emphasis on domain-specific objectives. Finally, Ludwig takes a declarative approach, letting users build deep-learning models by describing inputs and outputs rather than writing model code.

AutoML Platforms (Cloud)

Beyond open-source libraries sit commercial, managed platforms that deliver AutoML as a service. They handle the infrastructure, scaling, and deployment around automated model-building, and target organisations that want results without managing the underlying machinery. This section introduces the major offerings.

The large cloud providers each have their own service. Azure AutoML is Microsoft's offering within Azure Machine Learning, Vertex AI AutoML is Google Cloud's, and SageMaker Autopilot is the automated model-building component of Amazon's SageMaker. All three fit automated training and tuning into their broader machine-learning ecosystems, so models can move smoothly from experiment to production.

Alongside them are specialised vendors. DataRobot is one of the best-known enterprise AutoML platforms, emphasising governance and end-to-end automation. H2O Driverless AI focuses on automatic feature engineering and interpretability, while Dataiku AutoML embeds automated modelling inside a wider collaborative data-science platform used by analysts and engineers alike.

Meta-Learning

Most machine learning starts each new problem from scratch. Meta-learning asks a different question: can a system get better at learning itself, using experience from many past problems to master new ones faster? This research foundation underpins much of what makes automation possible.

The central idea is Meta-Learning, often described in plain terms as Learning to Learn — improving the learning process across many tasks rather than optimising for a single one. The component that carries this cross-task knowledge is the Meta-Learner, which observes how learning unfolds on many problems and adjusts how future learning is done. What it trains on is a Task Distribution (Meta-Learning) — a whole family of related tasks — so that skill transfers rather than being tied to one dataset.

A headline payoff is Few-Shot Meta-Learning, where a model can pick up a new task from only a handful of examples because it has learned how to learn efficiently. Two influential methods make this concrete. MAML (Model-Agnostic Meta-Learning) trains a model to a starting point from which it can adapt to a new task in just a few gradient steps, and Reptile (Meta-Learning) reaches a similar goal with a simpler, cheaper update rule.

Evaluation in AutoML

If a system is going to choose models automatically, how do we judge whether it is any good — and compare it fairly against alternatives? This section covers the notions and resources used to measure AutoML systems.

Because AutoML runs under real-world limits, two constraints shape every run. A Compute Budget (AutoML) caps how much processing power a search may consume, while a Time Budget (AutoML) caps how long it may run. These budgets make a particular quality important: Anytime Performance describes how good a system's best answer is if you stop it at any moment, rewarding methods that find decent models early and keep improving.

Fair comparison needs shared yardsticks. An AutoML Benchmark is a standard suite of datasets and rules for pitting systems against one another under matched conditions. Much of this work builds on OpenML, an open platform of datasets, tasks, and results that lets researchers reproduce and compare experiments across the community.

Applications

Automated machine learning is not an end in itself — its value shows up in what it lets people build and who gets to build it. This section surveys where AutoML is applied and the shift in access it represents.

Much of its impact is about people. The Democratization of ML is the broad trend of making model-building available beyond specialists, and its human face is the Citizen Data Scientist — a domain expert, such as a marketer or analyst, who can now build useful models without formal data-science training.

The technology adapts to the shape of the data. Tabular AutoML targets the spreadsheet-style data most organisations hold, automating modelling over rows and columns. Computer Vision AutoML brings the same automation to images, and NLP AutoML to text and language tasks. Data that unfolds over time gets dedicated treatment through Time Series AutoML, with AutoML for Forecasting focusing specifically on predicting future values such as demand, sales, or resource use.