Ordinary Least Squares (OLS)

When you draw a “best-fit” line through a cloud of points, you’re already thinking in the spirit of Ordinary Least Squares. OLS is the standard way to choose that line (or hyperplane) so predictions are as close as possible to the observed targets.

What OLS is doing

OLS fits a linear model of the form y = Xβ + ε, where X is your feature matrix, β are the coefficients you want to learn, and ε is noise. It chooses β to minimize the sum of squared residuals:

min_β  Σ_i (y_i - ŷ_i)^2

Squaring makes large errors count a lot, and it turns the problem into a clean optimization with a closed-form solution when XᵀX is invertible: β̂ = (XᵀX)⁻¹Xᵀy. Many libraries compute this with numerically stable decompositions rather than a literal matrix inverse.

How it shows up in real supervised tasks

House price prediction: features like square footage and location indicators map linearly to price.
Demand forecasting: price, promotions, and seasonality terms predict weekly sales.
Credit risk as regression: predicting expected loss amount (not just default vs. non-default).

In Python, you’ll meet OLS via scikit-learn’s LinearRegression or statsmodels’ OLS, which also reports coefficient uncertainty and hypothesis tests.

Why it matters (and what can go wrong)

OLS is the baseline for regression because it’s fast, interpretable, and often surprisingly strong. But it’s sensitive to outliers, struggles with multicollinearity (unstable coefficients), and can overfit when features are many—motivating regularized variants like Ridge and Lasso.

Ordinary Least Squares (OLS) is the standard method for fitting a linear regression model by choosing coefficients that minimize the sum of squared residuals between observed targets and the model’s predictions. Under common assumptions (e.g., linearity and unbiased errors), OLS yields unbiased, minimum-variance linear estimates and supports closed-form solutions and statistical inference. It matters because it is the baseline estimator for regression and the reference point for regularized variants.

Imagine you’re trying to draw the “best” straight line through a scatter of points on a chart—like matching house prices to house size. Ordinary Least Squares (OLS) is a common rule for choosing that line so it stays as close as possible to all the points.

“Close” here means the line tries to make the prediction errors small across the whole dataset. OLS doesn’t aim to be perfect for any one example; it aims to be the best overall fit.

In supervised learning, OLS is a basic way to train a linear regression model: you give it examples with known answers, and it learns a simple relationship you can use to predict new values.