Bias-Variance Tradeoff
When a model makes mistakes, it’s rarely just “bad at the task.” More commonly, it’s failing in one of two very different ways: it’s too rigid to learn the real pattern, or it’s so flexible that it learns noise as if it were signal.
What “bias” and “variance” mean
The bias-variance tradeoff is a way to explain prediction error by splitting it into two forces. Bias is error from overly simple assumptions: the model can’t represent the true relationship, so it makes systematic mistakes (think underfitting). Variance is error from being overly sensitive to the training data: small changes in the dataset lead to big changes in the learned model (think overfitting).
A useful mental picture: bias is like using the wrong map for a city (you’ll be consistently off), while variance is like redrawing the map every time you ask a different person (it changes too much).
How it shows up in real supervised tasks
In house price prediction, a straight-line model might miss nonlinear effects like neighborhood interactions: high bias. A very deep decision tree might perfectly match quirks of last year’s sales but fail on new listings: high variance. You’ll see this in diagnostics like learning curves:
- High bias: training error and validation error are both high and close together.
- High variance: training error is low, validation error is much higher.
Why it matters and how you control it
Most model choices are really bias-variance choices: model class, feature engineering, and regularization. For example, scikit-learn’s Ridge or Lasso increase bias slightly to reduce variance; limiting tree depth in XGBoost does the same. Cross-validation helps you pick the “sweet spot” where total generalization error is minimized.
The Bias-Variance Tradeoff is the fundamental tension between bias (systematic error from overly simple assumptions that causes underfitting) and variance (error from sensitivity to training data that causes overfitting) in a supervised model’s generalization error. Reducing one typically increases the other as model complexity changes. It matters because selecting model capacity, regularization, and data size aims to minimize total test error by balancing these two sources.
Think of throwing darts at a dartboard. If your darts land in a tight cluster but far from the bullseye, you’re being consistent but “off” in a systematic way. That’s bias. If your darts are scattered all over the board, you’re unpredictable from throw to throw. That’s variance.
In supervised learning, the bias-variance tradeoff is the balancing act between a model that’s too simple (often misses important patterns, like underestimating house prices) and a model that’s too complex (fits the training data too closely and then slips up on new data, like mislabeling new emails as spam). The goal is a sweet spot that generalizes well.