Residuals

When a regression model makes a prediction, the interesting part isn’t just the prediction—it’s the gap between what the model said and what actually happened. That gap is where you see what the model is missing, and it has a name: the residual.

What residuals are

In supervised regression, a residual is the difference between an observed target value and the model’s predicted value for the same example. For a data point i, it’s typically written as e_i = y_i − ŷ_i. Residuals can be positive (the model under-predicted) or negative (the model over-predicted). In ordinary least squares linear regression, the fitted line (or hyperplane) is chosen to minimize the sum of squared residuals, which is exactly what the common mean squared error (MSE) loss is built from.

Why residuals matter

Residuals are a practical diagnostic tool: they tell you whether the model’s errors look like “random noise” or like a pattern the model failed to capture. Key things residuals reveal include:

Nonlinearity: curved patterns in residuals vs. predictions suggest a linear model is too simple.
Heteroscedasticity: residual spread growing with the prediction suggests non-constant error variance.
Outliers/influential points: unusually large residuals can dominate the fit.

Concrete examples in regression work

In house-price prediction, consistently positive residuals for large homes can mean the model undervalues size at the high end. In demand forecasting, residuals that spike every weekend indicate missing calendar features. In scikit-learn, you can compute residuals directly after fitting LinearRegression:

y_pred = model.predict(X)
residuals = y - y_pred

Residuals are the per-example differences between the observed target value and a model’s predicted value (e.g., in linear regression, residual = y − ŷ). They quantify prediction error on the training data and are the quantities whose squared sum is minimized in ordinary least squares. Residuals matter because they drive parameter estimation, diagnostics (bias, outliers, heteroscedasticity), and uncertainty estimates; poorly behaved residuals signal model misspecification.

Imagine you’re trying to guess your friend’s arrival time, and you say “5:00 PM,” but they show up at 5:10. That 10-minute miss is like a residual: the gap between what you predicted and what actually happened.

In supervised learning, especially with linear regression, residuals are the “leftover errors” for each example in your data. If a model predicts a house will cost $400,000 but it sells for $420,000, the residual is $20,000. Looking at residuals helps you see how well the model is doing, spot patterns it’s missing, and notice unusual cases that don’t fit the trend.