Experiment
An experiment is any setup where the outcome isn’t fully predictable ahead of time, even if you repeat the same steps. In probability and statistics, it’s the “uncertainty-generating” process you’re trying to describe and learn from.
What “experiment” means technically
Formally, an experiment is a procedure that produces one result from a set of possible results. Those possible results are the sample space, and each run of the experiment produces an outcome. An event is a set of outcomes you care about (like “the customer churns” or “the test is positive”). A probability model assigns probabilities to events so you can reason about uncertainty in a consistent way.
Everyday examples that map cleanly to AI/ML
Think of experiments as “data-generating stories”:
- Medical test: outcome is positive/negative; event might be “positive given disease.”
- House sale: outcome is the sale price; event might be “price > $500k.”
- Ad click: outcome is click/no click; event might be “clicked by users under 25.”
Why it matters in machine learning
ML models quietly assume an experiment is producing your data. For example, supervised learning often treats each labeled example as a run of an experiment: features are observed, and a target outcome is generated. If the “experiment” changes—say, a new pricing policy, a sensor recalibration, or a different user population—your training and test data may no longer come from the same process. That breaks key assumptions behind evaluation and generalization, showing up as distribution shift or data drift.
Where you’ll see it in practice
Experiments show up in:
- A/B testing (randomized experiments for product decisions)
- Monte Carlo simulation (repeating a modeled experiment many times)
- Randomized algorithms and stochastic training (e.g., mini-batch sampling)
An Experiment is a well-defined procedure or data-generating process whose outcome is uncertain and modeled probabilistically. It specifies what is observed, under what conditions, and how randomness enters (e.g., sampling, noise, or stochastic environments). Experiments are central in AI/ML for defining likelihoods, estimating parameters, and evaluating models under controlled variation. Example: randomly splitting data into train/test sets is an experiment that induces random performance estimates.
Think of flipping a coin: you do the same action, but you don’t know exactly what you’ll get each time. That repeated “try” is an experiment.
In statistics and AI, an experiment means any setup where the outcome is uncertain, but the rules are clear. Rolling a die, measuring how long a website takes to load, or showing two versions of an ad to see which gets more clicks are all experiments. Each run gives an outcome, and by repeating the experiment many times, we can estimate how likely different outcomes are and make better predictions under uncertainty.