Notes
🔍

Multicollinearity

Sometimes your model is trying to learn the separate effect of two “different” inputs that are actually telling it almost the same thing. That situation—when predictors overlap heavily—is what multicollinearity is about.

What it is

Multicollinearity happens in regression-style models when two or more predictor variables are strongly linearly related (one can be approximated from the others). The model can still predict well, but it struggles to uniquely attribute changes in the outcome to each correlated predictor. In the extreme case (perfect collinearity), the regression coefficients aren’t even uniquely identifiable.

How it shows up in practice

Common real-world examples include:

  • House prices with both “square meters” and “number of rooms” as predictors.
  • Sales forecasting with “ad spend” and “impressions” (impressions largely driven by spend).
  • Health models with “weight,” “BMI,” and “waist circumference” together.

Typical symptoms are:

  • Unstable or counterintuitive coefficient signs (e.g., a positive factor appears negative).
  • Large standard errors and wide confidence intervals for coefficients.
  • Coefficients that change a lot when you add/remove a correlated feature.

Why it matters in AI/ML

In ML, multicollinearity is less about prediction accuracy and more about interpretability and reliability of learned weights. Linear regression, logistic regression, and linear SVMs can produce misleading feature importance when predictors are redundant. Regularized methods like ridge regression (L2) often handle it gracefully by shrinking correlated coefficients; lasso (L1) may pick one feature and drop others, which can be unstable when features are very similar. Tree-based models are usually less sensitive, but importance measures can still be “shared” across correlated features.

How it’s diagnosed

A common diagnostic is the Variance Inflation Factor (VIF), which quantifies how much a coefficient’s variance is inflated by correlation with other predictors. Pairwise correlations and condition numbers are also used.

Multicollinearity occurs in regression when two or more predictors are highly correlated, making it hard to separate their individual effects. It inflates coefficient standard errors, destabilizes estimates (small data changes cause big coefficient swings), and can make p-values and feature importance misleading, even if predictions remain accurate. Example: in a housing model, “square footage” and “number of rooms” may be so correlated that neither coefficient is reliably interpretable.

Imagine trying to figure out which ingredient makes a cake taste sweeter, but two ingredients always show up together—like sugar and honey in the same recipe. It becomes hard to tell which one is really responsible for the sweetness.

That’s what multicollinearity means in regression models: two or more input variables (features) are strongly related to each other. When this happens, the model can still predict fairly well, but it struggles to assign “credit” correctly to each variable. As a result, the estimated importance of individual variables can look unstable or misleading, making explanations and conclusions less trustworthy.