Guides / Probability

Variance, covariance and correlation: center first, divide by N − 1, and square the constants

Variance measures how far a quantity typically lands from its mean, covariance measures whether two quantities move together, and correlation rescales covariance to a number between −1 and 1. All three start the same way: subtract the mean before multiplying. With data, divide the sum of products by N − 1. The mistakes come from subtracting variances when the variables are subtracted, from scaling a variance by a instead of a², and from dividing a covariance by the variances instead of the standard deviations.

Why it works

Subtracting the mean puts every value on a scale where 0 is typical. A squared deviation is large whenever a value is far from the mean in either direction, so averaging the squares measures spread. For two variables, the product (x − x̄)(y − ȳ) is positive when both are on the same side of their means and negative when they are on opposite sides, so its average says whether they tend to move together. Covariance carries the units of both variables. Dividing by both standard deviations removes the units and gives the correlation, which by the Cauchy–Schwarz inequality can't leave [−1, 1].

The rules all come from expanding a square. Var(aX + bY) is the average of (a(X − μ_X) + b(Y − μ_Y))², which expands to a² times one squared deviation, b² times the other, and a cross term 2ab times the covariance. The data divide by N − 1 rather than N because the deviations are measured from x̄, which was fitted to the same data and sits slightly closer to them than the true mean does.

Example 1: sample variance and standard deviation

Find the sample variance and standard deviation of 3, 5, 7, 9.

1. x̄ = 24/4 = 6The mean first.
2. Σ(x − x̄)² = 9 + 1 + 1 + 9 = 20The deviations are −3, −1, 1, 3. They always add to 0, which is a check on x̄.
3. s² = 20/3 ≈ 6.667, s ≈ 2.582Divide by N − 1 = 3. Dividing by N gives 5, the population (or maximum-likelihood) version, which is what np.var returns by default.

Example 2: covariance and correlation of paired data

Find the covariance and correlation of x = 1, 2, 3, 4, 5 and y = 2, 4, 5, 4, 5.

1. x̄ = 3, ȳ = 20/5 = 4Center each variable on its own mean.
2. x − x̄ = −2, −1, 0, 1, 2; y − ȳ = −2, 0, 1, 0, 1Deviations, pair by pair.
3. Σ(x − x̄)(y − ȳ) = 4 + 0 + 0 + 0 + 2 = 6Multiply each pair, then add.
4. s_xy = 6/4 = 1.5Likewise s_x² = 10/4 = 2.5 and s_y² = 6/4 = 1.5, with the same N − 1 = 4 in all three.
5. r = 1.5/√(2.5 · 1.5) = 6/√60 ≈ 0.775The N − 1 cancels, so r is also 6 over √(10 · 6). A strong positive relationship, not a perfect line. The least-squares slope of y on x is 6/10 = 0.6.

Example 3: the variance of a combination

Var(X) = 4, Var(Y) = 1 and Cov(X, Y) = 1.2. Find Var(2X − 3Y + 5) and the correlation of X and Y.

1. a = 2, b = −3The constant 5 shifts every value and the mean together, so it drops out.
2. 4 · 4 + 9 · 1 + 2(2)(−3)(1.2)a² Var(X) + b² Var(Y) + 2ab Cov(X, Y).
3. = 16 + 9 − 14.4 = 10.6The cross term is negative because the coefficients have opposite signs while the covariance is positive.
4. corr(X, Y) = 1.2/(2 · 1) = 0.6Divide by the standard deviations √4 = 2 and √1 = 1.

Example 4: a covariance matrix and aᵀΣa

Four points are (2, 1), (4, 3), (6, 2), (8, 6). Find the sample covariance matrix, and use it to get the variance of x + y.

1. (−3, −2), (−1, 0), (1, −1), (3, 3)Subtract the means (5, 3) from every point.
2. Σ dx · dy = 6 + 0 − 1 + 9 = 14With Σ dx² = 20 and Σ dy² = 14, these are the three sums the matrix needs.
3. S = 1320141414Divide by N − 1 = 3. The variances sit on the diagonal and the covariance off it; S is symmetric.
4. a^TSa = (20 + 14 + 14 + 14)/3 = 62/3With a = (1, 1), Var(x + y) = a^TSa adds every entry of S: about 20.67. Check directly: the sums 3, 7, 8, 14 have mean 8 and squared deviations 25 + 1 + 0 + 36 = 62, over 3.

Common mistakes, and the exact line they happen on

1. Subtracting the variances for a difference

With Var(X) = 4, Var(Y) = 1 and Cov(X, Y) = 1.2 from Example 3, find Var(X − Y).

1. X − Y = 1 · X + (−1) · YRight so far: a = 1, b = −1.
2. Var(X − Y) = 4 − 1 = 3This is the mistake. The coefficient −1 is squared, so the variances add: b² = 1. Only the covariance term changes sign: Var(X − Y) = 4 + 1 − 2 · 1.2 = 2.6. A difference of two independent noisy quantities is noisier than either, never less noisy.

2. Scaling a variance by a instead of a²

The first term of Example 3.

1. Var(X) = 4Right so far.
2. Var(2X) = 2 · 4 = 8This is the mistake. Doubling X doubles every deviation, and the deviations are squared, so Var(2X) = 4 · 4 = 16. The standard deviation doubles, from 2 to 4; the variance quadruples.

3. Dividing the covariance by the variances

The correlation in Example 2.

1. s_xy = 1.5, s_x² = 2.5, s_y² = 1.5Right so far.
2. r = 1.5/(2.5 · 1.5) = 0.4This is the mistake. The denominator must have the same units as the covariance, which is the product of the standard deviations, not the variances. A quick test: rescaling x to x/10 shouldn't change a correlation, but it changes this number by a factor of 10. The right value is 1.5/√3.75 ≈ 0.775.

Practice

1. Find the sample variance of 2, 4, 6.

   Show answer

   Mean 4, squared deviations 4 + 0 + 4 = 8, so s² = 8/2 = 4.

2. Var(X) = 5. Find Var(3X + 2).

   Show answer

   3² · 5 = 45. The +2 changes nothing.

3. X and Y are independent with Var(X) = 2 and Var(Y) = 7. Find Var(X + Y) and Var(X − Y).

   Show answer

   Both are 9. With Cov(X, Y) = 0 the cross term vanishes, and the minus sign is squared away.

4. Cov(X, Y) = −3, and the standard deviations are 2 and 5. Find the correlation.

   Show answer

   r = −3/(2 · 5) = −0.3.

5. Find the covariance and correlation of x = 1, 2, 3 and y = 6, 4, 2.

   Show answer

   Deviations −1, 0, 1 and 2, 0, −2, so s_xy = −4/2 = −2, s_x = 1, s_y = 2 and r = −1: the points lie on the line y = 8 − 2x.