Guides / Probability

The multivariate Gaussian: invert Σ in the exponent, transform with AΣAᵀ, and condition with the Schur complement

The multivariate Gaussian describes several quantities at once through a mean vector μ and a covariance matrix Σ. Almost every calculation with it is one of four moves: evaluate the density through the Mahalanobis distance, push it through a linear map, condition on some coordinates, or draw a sample. The mistakes come from putting Σ in the exponent where Σ⁻¹ belongs, from dropping the covariance terms when combining coordinates, and from forgetting to center the observed value when conditioning.

Short answer

Density in d dimensions: p(x) = (2π)^−d/2 (det Σ)^−1/2 exp(−½ q), where q = (x − μ)^TΣ⁻¹(x − μ) is the squared Mahalanobis distance. Linear maps: if y = Ax + b, then y ~ N(Aμ + b, AΣA^T); a single coordinate is the case where A picks it out, so every marginal is Gaussian with its own mean and diagonal entry. Conditioning on x₂ in two dimensions: x₁ | x₂ ~ N(μ₁ + (Σ₁₂/Σ₂₂)(x₂ − μ₂), Σ₁₁ − Σ₁₂²/Σ₂₂). Sampling: factor Σ = LL^T (Cholesky) and set x = μ + Lε with ε standard normal.

Why it works

In one dimension the exponent is −(x − μ)²/(2σ²): the squared distance from the mean, measured in standard deviations. With several correlated coordinates, "measured in standard deviations" has to account for the shape of the cloud, and dividing by σ² becomes multiplying by Σ⁻¹. That is why q stretches distances along directions where the data are tight and shrinks them where the data are spread out. The factor (det Σ)^−1/2 plays the role of 1/σ: det Σ is the product of the eigenvalues of Σ, the variances along the principal axes, so its square root is the product of the standard deviations.

The linear-map rule is the covariance rule Cov(Ax) = AΣA^T, plus the fact that a linear combination of jointly Gaussian variables is again Gaussian. Conditioning is a regression: x₁ − (Σ₁₂/Σ₂₂)x₂ is uncorrelated with x₂, hence independent of it, so fixing x₂ shifts the mean along the regression line and leaves the leftover variance Σ₁₁ − Σ₁₂²/Σ₂₂, the Schur complement. Sampling works because Cov(Lε) = LIL^T = Σ.

Example 1: the density and the Mahalanobis distance

Let μ = (1, 2) and Σ = 4223. Find the squared Mahalanobis distance of x = (3, 2) from μ and the density there.

det Σ = 4 · 3 − 2 · 2 = 8Positive, as it must be for a covariance matrix.
Σ⁻¹ = 183−2−24Swap the diagonal, negate the off-diagonal, divide by the determinant.
x − μ = (2, 0)Subtract the mean.
q = (3 · 2 · 2)/8 = 1.5With a zero second entry only the (1, 1) entry of Σ⁻¹, 3/8, is used: q = (3/8) · 2².
√q ≈ 1.225The point is 2 units from the mean in plain distance, but only about 1.22 "standard deviations" once the spread of 4 along the first axis is accounted for.
p(x) = e^−0.75/(2π√8)The exponent is −q/2 = −0.75.
≈ 0.4724 × 0.05627e^−0.75 ≈ 0.4724 and 1/(2π√8) ≈ 0.05627.
p(x) ≈ 0.02658With d = 2 the constant is (2π)⁻¹(det Σ)^−1/2. The log-density is −ln 2π − ½ ln 8 − 0.75 ≈ −3.628.

Example 2: a linear map with AΣAᵀ

With the same μ and Σ, find the joint distribution of the sum s = x₁ + x₂ and the difference t = x₁ − x₂.

A = 111−1Each row of A is one new variable.
Aμ = (1 + 2, 1 − 2) = (3, −1)Means map straight through.
Var(s) = 4 + 3 + 2 · 2 = 11The diagonal of AΣA^T: each is a^TΣa for one row a, which includes the covariance twice.
Var(t) = 4 + 3 − 2 · 2 = 3The second row of A, with the covariance subtracted twice.
Cov(s, t) = Var(x₁) − Var(x₂)The cross terms Cov(x₁, x₂) cancel.
= 4 − 3 = 1The two variances.
(s, t) ~ N((3, −1), 11113)Reading off one coordinate gives a marginal: s alone is N(3, 11). Likewise x₁ alone is N(1, 4), with no integration needed.

Example 3: conditioning with the Schur complement

With the same μ and Σ, find the distribution of x₁ given that x₂ = 4.

x₂ − μ₂ = 4 − 2 = 2Center the observed value first: conditioning reacts to how surprising x₂ is, not to its raw size.
Σ₁₂/Σ₂₂ = 2/3The slope of the regression of x₁ on x₂.
mean = 1 + (2/3) · 2 = 7/3x₂ came out above its mean and the covariance is positive, so x₁ is pulled up too: 7/3 ≈ 2.333.
variance = 4 − 2²/3 = 8/3That is about 2.667, smaller than Σ₁₁ = 4: knowing x₂ removes some uncertainty. Two checks: it equals det Σ/Σ₂₂ = 8/3, and it is 1/(Σ⁻¹)₁₁ = 1/(3/8). It does not depend on the observed value 4 at all.

Example 4: sampling with a Cholesky factor

Turn the standard normal draw ε = (0.5, −1) into a sample from N(μ, Σ) with the same μ and Σ.

L₁₁ = √4 = 2, L₂₁ = 2/2 = 1The first column of L is the first column of Σ divided by √Σ₁₁.
L₂₂ = √(3 − 1²) = √2What is left of Σ₂₂ after the first column's share.
LL^T = 4223Check the factor before using it: 2 · 1 = 2 and 1 + 2 = 3.
(Lε)₁ = 2 · 0.5 = 1L is lower-triangular, so the first coordinate uses only ε₁.
(Lε)₂ = 1 · 0.5 + √2 · (−1)The second row of L uses both entries of ε …
= 0.5 − √2… so Lε = (1, 0.5 − √2).
x = (1 + 1, 2 + 0.5 − √2)Add the mean last.
x ≈ (2, 1.086)Since 2.5 − √2 ≈ 1.086. Repeating with many ε gives draws whose sample covariance approaches Σ.

Common mistakes, and the exact line they happen on

1. Σ in the exponent instead of Σ⁻¹

The quadratic form of Example 1.

x − μ = (2, 0)Right so far.
q = (x − μ)^TΣ(x − μ) = 16This is the mistake. This uses Σ itself, giving 4 · 2² = 16. In one dimension the exponent divides by σ², so the matrix version multiplies by the inverse. With Σ the density becomes e⁻⁸/(2π√8) ≈ 0.0000189 instead of 0.02658, and the error gets worse the more spread out the data are, which is backward. The right value is q = 1.5.

2. Dropping the covariance when combining coordinates

The variance of t = x₁ − x₂ in Example 2.

Var(x₁) = 4, Var(x₂) = 3Right so far: the diagonal of Σ.
Var(x₁ − x₂) = 4 + 3 = 7This is the mistake. Adding the variances is right only for uncorrelated coordinates. a^TΣa with a = (1, −1) uses every entry of Σ, including the off-diagonal 2 twice: 4 + 3 − 2 · 2 = 3. Positively correlated coordinates move together, so their difference varies less, not more.

3. Conditioning on the raw value instead of its deviation

The conditional mean of Example 3.

Σ₁₂/Σ₂₂ = 2/3Right so far.
mean = μ₁ + (2/3) · 4 = 11/3This is the mistake. The regression term must use x₂ − μ₂. A quick test: observing x₂ = μ₂ = 2 tells you nothing, so the conditional mean should stay at μ₁ = 1, but this formula gives 1 + 4/3. The right mean is 1 + (2/3)(4 − 2) = 7/3.

Practice

μ = (0, 0) and Σ = diag(4, 9). Find the squared Mahalanobis distance of x = (2, 3).
Show answer
With a diagonal Σ, q = 2²/4 + 3²/9 = 1 + 1 = 2: each coordinate is one standard deviation out.
Σ = 5222. Find det Σ and Var(x₁ + x₂).
Show answer
det Σ = 10 − 4 = 6, and Var(x₁ + x₂) = 5 + 2 + 2 · 2 = 11.
With μ = (0, 0) and the Σ of Problem 2, find the distribution of x₁ given x₂ = 1.
Show answer
Mean (2/2) · 1 = 1, variance 5 − 2²/2 = 3, so N(1, 3). Check: det Σ/Σ₂₂ = 6/2 = 3.
Find the Cholesky factor of 9335.
Show answer
L₁₁ = 3, L₂₁ = 3/3 = 1, L₂₂ = √(5 − 1) = 2, so L has rows (3, 0) and (1, 2).
Find the density at the mean of a 2-D Gaussian with Σ = diag(1, 4).
Show answer
At the mean q = 0, so p = 1/(2π√4) = 1/(4π) ≈ 0.0796.