Covariance

Covariance is a way to describe whether two quantities tend to move together. If house size goes up and price usually goes up too, you’d expect a positive covariance; if one rises while the other tends to fall, you’d expect a negative one.

What covariance measures
Technically, covariance measures how two variables vary together around their averages. For variables X and Y, it looks at the product of their “deviations from the mean” and averages those products:

Positive covariance: when X is above its mean, Y tends to be above its mean too (they move in the same direction).
Negative covariance: when X is above its mean, Y tends to be below its mean (they move in opposite directions).
Near zero covariance: no consistent linear co-movement (though non-linear relationships can still exist).

A key detail: covariance has units (e.g., square-meters × dollars), so its magnitude depends on the scale of the variables.

Covariance vs. correlation
Because covariance depends on units, it’s hard to compare across different pairs of variables. Correlation fixes this by scaling covariance by the variables’ standard deviations, producing a unit-free number between -1 and 1. Covariance is the raw “co-movement”; correlation is the standardized version.

Why it matters in AI/ML
Covariance shows up all over machine learning:

Covariance matrices summarize how features move together; they’re central to PCA, where directions of maximum variance are found via eigenvectors of the covariance matrix.
In Gaussian models (e.g., multivariate normal, Gaussian processes), covariance controls the shape and orientation of probability “clouds.”
Feature dependence (high covariance) can signal multicollinearity, which can make linear model coefficients unstable.

In practice, libraries like NumPy compute it with functions such as numpy.cov.

Covariance measures how two variables vary together: it is positive when they tend to increase together, negative when one increases as the other decreases, and near zero when their linear co-variation is weak. Its magnitude depends on the variables’ scales, so it is often normalized into correlation. In AI/ML, covariance underlies covariance matrices used in PCA, Gaussian models, and feature whitening; e.g., PCA finds directions with maximum covariance.

Imagine you’re watching two things at once: how much people study and what scores they get. If, on days someone studies more, their score also tends to be higher, those two things “move together.” Covariance is a number that captures this idea of moving together.

If the covariance is positive, the two values often rise and fall together (more study, higher score). If it’s negative, one tends to go up when the other goes down (more speed, less reaction time). If it’s near zero, there’s no clear pattern. In AI and machine learning, covariance helps describe relationships between features and is used in tools like covariance matrices.