Guides / Linear algebra

Principal component analysis: center the data, take the eigenvectors of the covariance, and keep the top ones

Principal component analysis (PCA) finds the directions along which a cloud of data points spreads the most. The recipe is short: subtract the mean, build the covariance matrix, and find its eigenvectors. The eigenvectors are the principal directions, the eigenvalues are the variances along them, and keeping the top few gives the best low-dimensional summary of the data. The mistakes come from skipping the centering, from forgetting to add the mean back when reconstructing, and from reading the explained variance off singular values without squaring them.

Why it works

For a unit vector u, the numbers u^T(x_i − x̄) are the positions of the points along the line through x̄ in the direction u, and their variance is u^TSu. Writing u in the orthonormal eigenvector basis of S turns this into a weighted average of the eigenvalues, λ₁c₁² + λ₂c₂² + … with c₁² + c₂² + … = 1. A weighted average is largest when all the weight sits on the biggest value, so the direction of greatest spread is q₁, and the spread there is λ₁. Among directions perpendicular to q₁, the best is q₂, and so on.

The same picture explains the error. Each centered point splits into its components along the eigenvectors, and those pieces are perpendicular, so squared lengths add (Pythagoras). Keeping the first k components and dropping the rest leaves an error made of exactly the dropped pieces, whose average squared size is the sum of the dropped eigenvalues. The total variance, the trace of S, is the sum of all the eigenvalues, which is why explained-variance ratios add to 1. Dividing by N instead of N − 1 scales every eigenvalue by the same factor and changes none of the directions or ratios.

Example 1: center the data and build the covariance matrix

Four points are (0, 0), (3, 3), (6, 3) and (7, 2). Find the mean and the sample covariance matrix.

1. x̄ = (16/4, 8/4) = (4, 2)Average each coordinate.
2. (−4, −2), (−1, 1)Subtract x̄ from every point: the first two …
3. (2, 1), (3, 0)… and the last two. Each column of deviations adds to 0, which checks the mean.
4. Σ dx² = 30The first of the three sums a 2 × 2 covariance matrix needs: 16 + 1 + 4 + 9.
5. Σ dy² = 64 + 1 + 1 + 0.
6. Σ dx · dy = 98 − 1 + 2 + 0.
7. S = 10332Divide each sum by N − 1 = 3. The variances 10 and 2 sit on the diagonal, the covariance 3 off it, and S is symmetric.

Example 2: principal directions and explained variance

Find the principal directions of the data in Example 1, the variance along each, and the fraction of the total variance the first component explains.

1. λ² − 12λ + 11 = 0det(S − λI) = (10 − λ)(2 − λ) − 9, with trace 12 and determinant 20 − 9 = 11.
2. λ₁ = 11, λ₂ = 1The roots add to the trace and multiply to the determinant.
3. λ = 11: −x + 3y = 0The first row of S − 11I is (−1, 3).
4. q₁ = (3, 1)/√10Normalize, since scores need unit vectors.
5. q₂ = (1, −3)/√10A symmetric matrix has perpendicular eigenvectors, so the second direction is q₁ turned by 90°. Check: S(1, −3) = (1, −3).
6. 11/(11 + 1) = 11/12 ≈ 91.7%The first component explains most of the spread. Its direction makes an angle of about 18.4° with the x-axis, and the variance along it, 11, beats the 10 along the x-axis itself.

Example 3: scores, reconstruction and the error

Using one component, find the scores of the four points of Example 1, reconstruct (0, 0), and find the mean squared reconstruction error.

1. z = (3 dx + dy)/√10Dot each centered point with q₁.
2. = (−14, −2, 7, 9)/√10The scores add to 0, as the centered data do.
3. (196 + 4 + 49 + 81)/10 = 33The sum of the squared scores.
4. 33/3 = 11Divide by N − 1. The variance of the scores is λ₁, a check on Example 2.
5. x̂ = (4, 2) + (−14/√10)q₁Start at the mean and walk along q₁ by the score.
6. = (4, 2) − 1.4(3, 1)q₁ = (3, 1)/√10, and the two √10's multiply to 10.
7. x̂ = (−0.2, 0.6)The reconstruction is the foot of the perpendicular from (0, 0) to the line through the mean along q₁.
8. error (0.2, −0.6)The point (0, 0) minus x̂.
9. squared length 0.40.04 + 0.36.
10. (0.4 + 1.6 + 0.1 + 0.9)/3The squared errors of all four points, divided by N − 1 like the variances …
11. = 1 = λ₂… add up to the discarded eigenvalue, exactly as the short answer says.

Example 4: how many components, from an SVD

The centered data matrix of N = 11 points in four dimensions has singular values 20, 10, 5 and 2. How many components are needed to explain at least 95% of the variance?

1. λ_j = σ_j²/(N − 1)If X_c = UΣV^T, then X_c^TX_c = VΣ²V^T, so the right singular vectors are the principal directions and the squared singular values, over N − 1, are the variances.
2. λ = 40, 10, 2.5, 0.4The four variances.
3. total = 52.9The sum of the variances, the trace of S.
4. 40/52.9 ≈ 75.6%Cumulative explained variance after one component …
5. 50/52.9 ≈ 94.5%… after two …
6. 52.5/52.9 ≈ 99.2%… and after three.
7. k = 3Two components fall just short of 95%; three clear it. The N − 1 cancels from every ratio, so 400/529, 500/529 and 525/529 give the same percentages.

Common mistakes, and the exact line they happen on

1. Skipping the centering

The data of Example 1, with the mean left in.

1. X^TX = 94414122The sums of x², xy and y² over the raw points (0, 0), (3, 3), (6, 3), (7, 2).
2. q₁ ≈ (0.911, 0.412)This is the mistake. This top eigenvector points at about 24.4°. Without centering, the matrix measures spread around the origin, so the top direction is pulled toward the mean (4, 2), which sits at 26.6°. The true principal direction is at 18.4°, and the variance of the centered data along this wrong direction is about 10.89, less than λ₁ = 11. The uncentered “explained variance” of about 97% is just as misleading. Subtract x̄ first.

2. Reconstructing without adding the mean back

The reconstruction of (0, 0) in Example 3.

1. z₁ = −14/√10Right so far: the score was computed from the centered point.
2. x̂ = z₁q₁ = (−4.2, −1.4)This is the mistake. z₁q₁ is the reconstruction of the centered point. Every reconstruction made this way is off by exactly the mean (4, 2). Add x̄ back: x̂ = (−0.2, 0.6).

3. Explained variance from unsquared singular values

The singular values 20, 10, 5, 2 of Example 4.

1. σ = 20, 10, 5, 2Right so far: these are what an SVD routine returns.
2. 20/37 ≈ 54.1%This is the mistake. Singular values are standard deviations (times √(N − 1)), not variances. Read this way, three components fall short of 95% and all four seem needed. Square first: 400, 100, 25 and 4 give 75.6%, 94.5% and 99.2%, so three components are enough.
3. 30/37 ≈ 81.1%The same misreading after two components …
4. 35/37 ≈ 94.6%… and after three, just short of 95%.

Practice

1. A covariance matrix is diagonal with entries 6 and 2. What are the principal directions, and what fraction of the variance does the first component explain?

   Show answer

   The coordinate axes, (1, 0) and then (0, 1), because a diagonal matrix's eigenvectors are the axes. The first explains 6/8 = 75%.

2. Find the eigenvalues and the first principal direction of S = 3113.

   Show answer

   λ = 4 and 2; q₁ = (1, 1)/√2. The variance along the diagonal, 4, is more than the 3 along either axis.

3. The mean is (1, 2) and the first principal direction is (0.6, 0.8). Find the score of the point (4, 5), its one-component reconstruction, and the reconstruction error.

   Show answer

   Centered point (3, 3), score 0.6 · 3 + 0.8 · 3 = 4.2, reconstruction (1, 2) + 4.2(0.6, 0.8) = (3.52, 5.36), error vector (0.48, −0.36) of length 0.6.

4. A covariance matrix has eigenvalues 9, 4, 1 and 1. Keeping two components, what fraction of the variance is kept, and what is the mean squared reconstruction error?

   Show answer

   13/15 ≈ 86.7% kept; the error is the discarded 1 + 1 = 2.

5. The centered data matrix of N = 5 points has singular values 6 and 2. Find the variances along the principal directions and the fraction explained by the first.

   Show answer

   36/4 = 9 and 4/4 = 1, so the first explains 9/10 = 90%. Using 6/8 = 75% forgets to square.