The bias–variance decomposition

Worked solutions

Problem 1

1. Let $m = \mathbb{E}[\hat\theta]$. Then $\hat\theta - \theta = (\hat\theta - m) + (m - \theta)$.Add and subtract the mean of the estimator; the second bracket is the bias, a constant.
2. $(\hat\theta - \theta)^2 = (\hat\theta - m)^2 + 2(\hat\theta - m)(m - \theta) + (m - \theta)^2$.Expand the square.
3. $\mathbb{E}[(\hat\theta - m)(m - \theta)] = (m - \theta)\,\mathbb{E}[\hat\theta - m] = (m - \theta)\cdot 0$.$m - \theta$ is a constant and comes out; $\mathbb{E}[\hat\theta] - m = 0$ by the definition of $m$.
4. $\mathbb{E}[(\hat\theta - \theta)^2] = \mathbb{E}[(\hat\theta - m)^2] + (m - \theta)^2 = \operatorname{Var}(\hat\theta) + \operatorname{bias}(\hat\theta)^2$.Take expectations of step 2; the cross term vanishes by step 3; the remaining terms are the definitions.
5. $\operatorname{MSE}(\hat\theta) = \operatorname{bias}(\hat\theta)^2 + \operatorname{Var}(\hat\theta)$The error splits into a systematic part, how far the average estimate sits from the truth, and a random part, how much the estimate moves from sample to sample, and the two add because the cross term averages out. Nothing about the distribution was used, only that $\theta$ is fixed and $\hat\theta$ has a mean and a variance. The decomposition is an identity, not an inequality, so lowering the MSE means lowering the sum, and Problems 2, 6 and 9 all do it by accepting some bias for a larger drop in variance; Mistake 1 refuses the trade.

Problem 2

1. $\mathbb{E}[\bar x] = \tfrac1N\sum_n\mathbb{E}[x_n] = \mu$, so $\operatorname{bias}(\bar x) = 0$.Linearity of expectation; every $x_n$ has mean $\mu$.
2. $\operatorname{Var}(\bar x) = \tfrac1{N^2}\sum_n\operatorname{Var}(x_n) = \dfrac{\sigma^2}N$, so $\operatorname{MSE}(\bar x) = \dfrac{\sigma^2}N$.Independence makes the variance of the sum the sum of the variances; $\operatorname{Var}(aX) = a^2\operatorname{Var}(X)$ with $a = 1/N$; Problem 1 with zero bias.
3. $\mathbb{E}[c\bar x] = c\mu$, so $\operatorname{bias}(c\bar x) = (c - 1)\mu$; $\operatorname{Var}(c\bar x) = c^2\sigma^2/N$.Scaling by the constant $c$ scales the mean by $c$ and the variance by $c^2$.
4. $\operatorname{MSE}(c\bar x) = (1 - c)^2\mu^2 + c^2\dfrac{\sigma^2}N$.Problem 1.
5. $\dfrac{d}{dc}\operatorname{MSE} = -2(1 - c)\mu^2 + 2c\dfrac{\sigma^2}N = 0 \iff c^* = \dfrac{\mu^2}{\mu^2 + \sigma^2/N}$, a minimum since the second derivative $2\mu^2 + 2\sigma^2/N > 0$.Differentiate the quadratic in $c$ and solve; it is convex, so the stationary point is the minimum.
6. $\mu = 1$, $\sigma^2 = 1$, $N = 4$: $c^* = \dfrac{1}{1 + 1/4} = 0.8$, $\operatorname{MSE}(0.8\bar x) = 0.04\cdot 1 + 0.64\cdot 0.25 = 0.2 < 0.25 = \operatorname{MSE}(\bar x)$.Step 5, then step 4 with $c = 0.8$: bias $-0.2$, variance $0.16$.
7. $\bar x$: bias $0$, variance $\sigma^2/N$, MSE $\sigma^2/N$; $c\bar x$: bias $(c - 1)\mu$, variance $c^2\sigma^2/N$, MSE $(1 - c)^2\mu^2 + c^2\sigma^2/N$, minimised at $c^* = \dfrac{\mu^2}{\mu^2 + \sigma^2/N}$; for $\mu = 1$, $\sigma^2 = 1$, $N = 4$: $c^* = 0.8$ and MSE $0.2$ against $0.25$Shrinking towards zero trades a bias of $0.2$ for a variance cut from $0.25$ to $0.16$, and wins. The catch is that $c^*$ depends on $\mu$, the thing being estimated, so it is not an estimator one can use directly; it says the unbiased choice is not the floor, and that any prior knowledge of the size of $\mu$ (here, that $\mu^2$ is comparable to $\sigma^2/N$) can be spent. Ridge regression in Problem 6 is the same calculation with the same answer, and the Bayesian posterior mean under a Gaussian prior lands on exactly $c^*\bar x$ with $\mu^2$ replaced by the prior variance.

Problem 3

1. $y - \hat f_D(x) = \varepsilon + \big(f(x) - \hat f_D(x)\big)$.Substitute $y = f(x) + \varepsilon$ and regroup.
2. $(y - \hat f_D(x))^2 = \varepsilon^2 + 2\varepsilon\big(f(x) - \hat f_D(x)\big) + \big(f(x) - \hat f_D(x)\big)^2$.Expand the square.
3. $\mathbb{E}[\varepsilon^2] = \sigma^2$ and $\mathbb{E}\big[\varepsilon\,(f(x) - \hat f_D(x))\big] = \mathbb{E}[\varepsilon]\,\mathbb{E}_D[f(x) - \hat f_D(x)] = 0$.$\varepsilon$ has mean zero and variance $\sigma^2$; the test noise is independent of the training set $D$, so the expectation of the product is the product of the expectations (the variance page).
4. $\mathbb{E}_D\big[(f(x) - \hat f_D(x))^2\big] = \big(\mathbb{E}_D[\hat f_D(x)] - f(x)\big)^2 + \operatorname{Var}_D\big(\hat f_D(x)\big)$.Problem 1 with $\hat\theta = \hat f_D(x)$, an estimator of the fixed number $\theta = f(x)$.
5. $\mathbb{E}\big[(y - \hat f_D(x))^2\big] = \sigma^2 + \operatorname{bias}^2 + \operatorname{variance}$, with $\operatorname{bias} = \mathbb{E}_D[\hat f_D(x)] - f(x)$ and $\operatorname{variance} = \operatorname{Var}_D(\hat f_D(x))$; the $\sigma^2$ cannot be reducedThe expected test error at a point has a floor, the noise variance, that no learner reaches: even $\hat f_D = f$ exactly scores $\sigma^2$ (Mistake 4). Above the floor sit the two terms of Problem 1 for the learner regarded as an estimator of $f(x)$; averaging over $x$ gives the usual statement about test error. Step 3 used that the test noise is independent of $D$, and that is what fails on the training set, where the same $\varepsilon_n$ sits in $y_n$ and in $\hat f_D$ (Mistake 2). The check evaluates the left side exactly for the linear model of Problems 4 and 5, where $\hat f_D(x) = \hat w\,x$ is Gaussian, and compares it with the three terms.

Problem 4

1. $\sum_nx_ny_n = \sum_nx_n(wx_n + \varepsilon_n) = ws + \sum_nx_n\varepsilon_n$.Substitute the model and use $\sum_nx_n^2 = s$.
2. $\hat w = \dfrac{ws + \sum_nx_n\varepsilon_n}{s} = w + \dfrac{\sum_nx_n\varepsilon_n}{s}$.Divide by $s$.
3. $\mathbb{E}[\hat w] = w + \dfrac{\sum_nx_n\mathbb{E}[\varepsilon_n]}{s} = w$.The $x_n$ are fixed and every $\varepsilon_n$ has mean zero.
4. $\operatorname{Var}(\hat w) = \dfrac{1}{s^2}\sum_nx_n^2\operatorname{Var}(\varepsilon_n) = \dfrac{\sigma^2s}{s^2} = \dfrac{\sigma^2}s$.Independent noise: the variance of $\sum_nx_n\varepsilon_n$ is $\sum_nx_n^2\sigma^2$; then $\operatorname{Var}(aX) = a^2\operatorname{Var}(X)$ with $a = 1/s$.
5. $\hat w = w + \sum_nx_n\varepsilon_n/s$; $\operatorname{bias} = 0$; $\operatorname{Var}(\hat w) = \sigma^2/s$, so $\operatorname{MSE}(\hat w) = \sigma^2/s$The estimator is the truth plus a weighted average of the noise, and its variance falls as $s = \sum_nx_n^2$ grows: more points, or points farther from the origin, pin the slope down. This is the scalar case of the least-squares page's $\operatorname{Cov}(\hat w) = \sigma^2(X^\top X)^{-1}$, with $X^\top X = s$. The regression page shows that least squares is also the maximum-likelihood estimator under Gaussian noise, and the MLE page's variance estimator is biased for the same structural reason that Problem 9 revisits; the slope estimator itself is not.

Problem 5

1. $\hat w_\lambda = \dfrac{\sum_nx_ny_n}{s + \lambda} = \dfrac{s}{s + \lambda}\cdot\dfrac{\sum_nx_ny_n}{s} = \dfrac{s}{s + \lambda}\hat w$.Multiply and divide by $s$.
2. $\mathbb{E}[\hat w_\lambda] = \dfrac{s}{s + \lambda}w$, so $\operatorname{bias}(\hat w_\lambda) = \dfrac{s}{s + \lambda}w - w = -\dfrac{\lambda}{s + \lambda}w$.Problem 4's $\mathbb{E}[\hat w] = w$ scaled by the constant; $\tfrac{s}{s + \lambda} - 1 = -\tfrac{\lambda}{s + \lambda}$.
3. $\operatorname{Var}(\hat w_\lambda) = \Big(\dfrac{s}{s + \lambda}\Big)^2\dfrac{\sigma^2}s = \dfrac{\sigma^2s}{(s + \lambda)^2}$.$\operatorname{Var}(aX) = a^2\operatorname{Var}(X)$ with Problem 4's variance.
4. $\operatorname{MSE}(\hat w_\lambda) = \dfrac{\lambda^2w^2}{(s + \lambda)^2} + \dfrac{\sigma^2s}{(s + \lambda)^2} = \dfrac{\lambda^2w^2 + \sigma^2s}{(s + \lambda)^2}$.Problem 1; common denominator.
5. $\hat w_\lambda = \dfrac{s}{s + \lambda}\hat w$; $\operatorname{bias} = -\dfrac{\lambda w}{s + \lambda}$; $\operatorname{Var} = \dfrac{\sigma^2s}{(s + \lambda)^2}$; $\operatorname{MSE} = \dfrac{\lambda^2w^2 + \sigma^2s}{(s + \lambda)^2}$Ridge is Problem 2's shrinkage with $c = s/(s + \lambda)$: the estimate is pulled towards zero by a factor that the data size $s$ and the penalty $\lambda$ fight over, the bias is the part of $w$ lost to the pull, and the variance is cut by $c^2$. At $\lambda = 0$ everything reduces to Problem 4; as $\lambda \to \infty$ the estimate goes to $0$, with bias $-w$ and no variance. The variance is $\sigma^2s/(s + \lambda)^2$ and not $\sigma^2/(s + \lambda)$ (Mistake 3), a distinction that matters for everything in Problems 6 and 7.

Problem 6

1. $\operatorname{MSE}(\lambda) = \dfrac{u(\lambda)}{v(\lambda)}$ with $u = \lambda^2w^2 + \sigma^2s$ and $v = (s + \lambda)^2$; $u' = 2\lambda w^2$ and $v' = 2(s + \lambda)$.Name the pieces for the quotient rule.
2. $\dfrac{d}{d\lambda}\operatorname{MSE} = \dfrac{u'v - uv'}{v^2} = \dfrac{2(s + \lambda)\big[\lambda w^2(s + \lambda) - \lambda^2w^2 - \sigma^2s\big]}{(s + \lambda)^4} = \dfrac{2s\,(\lambda w^2 - \sigma^2)}{(s + \lambda)^3}$.Quotient rule; factor $2(s + \lambda)$ from the numerator; inside the bracket $\lambda w^2s + \lambda^2w^2 - \lambda^2w^2 - \sigma^2s = s(\lambda w^2 - \sigma^2)$.
3. The derivative is negative for $\lambda < \sigma^2/w^2$ and positive for $\lambda > \sigma^2/w^2$, so $\lambda^* = \dfrac{\sigma^2}{w^2}$ is the minimum.The sign of step 2 is the sign of $\lambda w^2 - \sigma^2$; the function decreases then increases.
4. $\operatorname{MSE}(\lambda^*) = \dfrac{\sigma^4/w^2 + \sigma^2s}{(s + \sigma^2/w^2)^2} = \dfrac{\sigma^2(s + \sigma^2/w^2)}{(s + \sigma^2/w^2)^2} = \dfrac{\sigma^2}{s + \sigma^2/w^2}$.Substitute $\lambda^*$; the numerator is $\sigma^2$ times the base of the denominator.
5. $\lambda^* = \sigma^2/w^2 > 0$ whenever $\sigma^2 > 0$, and $\dfrac{\sigma^2}{s + \sigma^2/w^2} < \dfrac{\sigma^2}s = \operatorname{MSE}(\hat w)$.A positive number divided by $w^2$; a larger denominator.
6. $w = 1$, $\sigma^2 = 1$, $s = 4$: $\lambda^* = 1$, $\operatorname{MSE}(\lambda^*) = \dfrac1{4 + 1} = 0.2$ against $\operatorname{MSE}(\hat w) = 0.25$; at $\lambda^* = 1$ the bias is $-0.2$ and the variance $4/25 = 0.16$.Steps 3 and 4; Problem 5's bias and variance at $\lambda = 1$.
7. $\lambda^* = \sigma^2/w^2$ with $\operatorname{MSE}(\lambda^*) = \dfrac{\sigma^2}{s + \sigma^2/w^2} < \dfrac{\sigma^2}s$; for $w = 1$, $\sigma^2 = 1$, $s = 4$: $\lambda^* = 1$, MSE $0.2$ against $0.25$The best penalty is the noise-to-signal ratio $\sigma^2/w^2$, and it does not depend on $s$: more data does not change the right $\lambda$, it makes the shrinkage factor $s/(s + \lambda)$ closer to $1$ so that the penalty matters less. The numbers are Problem 2's exactly, because $\hat w_\lambda = c\hat w$ with $c = s/(s + \lambda)$ and $\lambda^*$ corresponds to $c^* = w^2/(w^2 + \sigma^2/s)$. In the Bayesian reading, $\lambda = \sigma^2/\tau^2$ is the ridge penalty that makes the posterior mean under the prior $w \sim \mathcal{N}(0, \tau^2)$, and $\lambda^*$ is the prior whose variance matches $w^2$. As in Problem 2, $\lambda^*$ needs $w$, so in practice it is chosen by cross-validation, which estimates the MSE curve of Problem 7 from the data.

Problem 7

1. $\operatorname{bias}^2 = \dfrac{\lambda^2w^2}{(s + \lambda)^2}$ and $\dfrac{d}{d\lambda}\operatorname{bias}^2 = w^2\cdot\dfrac{2\lambda(s + \lambda)^2 - \lambda^2\cdot 2(s + \lambda)}{(s + \lambda)^4} = \dfrac{2\lambda sw^2}{(s + \lambda)^3} > 0$ for $\lambda > 0$.Quotient rule; factor $2\lambda(s + \lambda)$ from the numerator, leaving $(s + \lambda) - \lambda = s$.
2. $\operatorname{Var} = \dfrac{\sigma^2s}{(s + \lambda)^2}$ and $\dfrac{d}{d\lambda}\operatorname{Var} = -\dfrac{2\sigma^2s}{(s + \lambda)^3} < 0$.Power rule on $(s + \lambda)^{-2}$.
3. As $\lambda \to \infty$: $\operatorname{bias}^2 \to w^2$ and $\operatorname{Var} \to 0$.$\lambda^2/(s + \lambda)^2 \to 1$ and $s/(s + \lambda)^2 \to 0$.
4. $\operatorname{bias}^2 = \operatorname{Var} \iff \lambda^2w^2 = \sigma^2s \iff \lambda = \dfrac{\sigma\sqrt s}{w}$.Equate the numerators, which share the denominator; take the positive root.
5. $\dfrac{d}{d\lambda}\operatorname{bias}^2 = \dfrac{2\lambda sw^2}{(s + \lambda)^3} > 0$, $\dfrac{d}{d\lambda}\operatorname{Var} = -\dfrac{2\sigma^2s}{(s + \lambda)^3} < 0$; limits $w^2$ and $0$; the terms are equal at $\lambda = \sigma\sqrt s/w$This is the picture behind every regularisation plot: a rising bias curve, a falling variance curve and a U-shaped sum. Adding the two derivatives gives $\dfrac{2s(\lambda w^2 - \sigma^2)}{(s + \lambda)^3}$, Problem 6's derivative, and the sum is minimised where the two slopes cancel, at $\lambda^* = \sigma^2/w^2$, which is not where the curves cross: for Problem 6's numbers they cross at $\lambda = 2$ while the minimum is at $\lambda = 1$, where the variance term ($0.16$) is still four times the squared bias ($0.04$). The same shape appears with model size, training time and the number of neighbours (Problem 10) as the knob.

Problem 8

1. $\mathbb{E}[\bar f] = \tfrac1B\sum_b\mathbb{E}[\hat f_b] = \tfrac1B\cdot Bm = m$.Linearity; every predictor has the same mean. The bias $m - f(x)$ is therefore the same as each member's.
2. Let $z = (\hat f_1, \dots, \hat f_B)^\top$ with covariance $\Sigma$, $\Sigma_{bb} = v$ and $\Sigma_{bb'} = \rho v$ for $b \neq b'$. Then $\bar f = \tfrac1B\mathbf{1}^\top z$ and $\operatorname{Var}(\bar f) = \tfrac1{B^2}\mathbf{1}^\top\Sigma\mathbf{1}$.The variance page's $\operatorname{Cov}(Az) = A\Sigma A^\top$ with the $1\times B$ matrix $A = \tfrac1B\mathbf{1}^\top$.
3. $\mathbf{1}^\top\Sigma\mathbf{1} = \sum_{b,b'}\Sigma_{bb'} = Bv + B(B - 1)\rho v$.$B$ diagonal entries equal to $v$ and $B(B - 1)$ off-diagonal entries equal to $\rho v$.
4. $\operatorname{Var}(\bar f) = \dfrac{Bv + B(B - 1)\rho v}{B^2} = \dfrac{v + (B - 1)\rho v}B = \rho v + \dfrac{(1 - \rho)v}B$.Divide by $B^2$; split $(B - 1)\rho v = B\rho v - \rho v$.
5. $\mathbb{E}[\bar f] = m$ (bias unchanged); $\operatorname{Var}(\bar f) = \rho v + \dfrac{(1 - \rho)v}B$; for $\rho = 0$ it is $v/B$, and as $B \to \infty$ it tends to $\rho v$Averaging models attacks only the variance term of Problem 3, and only the part of it that is not shared: with independent members the variance falls like $1/B$ (the variance page's sample mean again), and with correlated members it stops at the floor $\rho v$ however many are averaged (Mistake 5). Bagging trains members on bootstrap resamples to make them differ, and random forests decorrelate them further by restricting each split to a random subset of features, lowering $\rho$ at the cost of a little bias; the Jensen page's ambiguity decomposition is the same conclusion reached for one dataset rather than in expectation. Boosting is the opposite strategy: it attacks the bias term by fitting each new model to the residual of the current average.

Problem 9

1. $\mathbb{E}[S] = (N - 1)\sigma^2$ and $\operatorname{Var}(S) = 2(N - 1)\sigma^4$.$S/\sigma^2$ is chi-square with $N - 1$ degrees of freedom, with mean $N - 1$ and variance $2(N - 1)$; scaling by $\sigma^2$ scales the mean by $\sigma^2$ and the variance by $\sigma^4$.
2. $\operatorname{bias}(cS) = c(N - 1)\sigma^2 - \sigma^2 = \big(c(N - 1) - 1\big)\sigma^2$ and $\operatorname{Var}(cS) = 2c^2(N - 1)\sigma^4$.Scale step 1 by $c$; subtract the target $\sigma^2$.
3. $\operatorname{MSE}(cS) = \sigma^4\Big[\big(c(N - 1) - 1\big)^2 + 2c^2(N - 1)\Big]$.Problem 1.
4. $\dfrac{d}{dc}\operatorname{MSE} = \sigma^4\Big[2(N - 1)\big(c(N - 1) - 1\big) + 4c(N - 1)\Big] = 2(N - 1)\sigma^4\big[c(N - 1) - 1 + 2c\big] = 2(N - 1)\sigma^4\big[c(N + 1) - 1\big]$.Differentiate the quadratic in $c$; collect the terms in $c$.
5. $c^* = \dfrac1{N + 1}$, a minimum since the MSE is a convex quadratic in $c$; there $\operatorname{bias} = -\dfrac{2\sigma^2}{N + 1}$, $\operatorname{Var} = \dfrac{2(N - 1)\sigma^4}{(N + 1)^2}$ and $\operatorname{MSE} = \dfrac{\sigma^4\big(4 + 2(N - 1)\big)}{(N + 1)^2} = \dfrac{2\sigma^4}{N + 1}$.Set step 4 to zero; substitute $c^*$ into step 2 and add.
6. $c = \tfrac1{N - 1}$: bias $0$, $\operatorname{MSE} = \dfrac{2\sigma^4}{N - 1}$. $c = \tfrac1N$: bias $-\dfrac{\sigma^2}N$, $\operatorname{MSE} = \sigma^4\Big[\dfrac1{N^2} + \dfrac{2(N - 1)}{N^2}\Big] = \dfrac{(2N - 1)\sigma^4}{N^2}$.Step 3 at the two values; $(\tfrac{N - 1}N - 1)^2 = \tfrac1{N^2}$.
7. $N = 5$, in units of $\sigma^4$: $S/4$ gives $\tfrac24 = 0.5$; $S/5$ gives $\tfrac9{25} = 0.36$; $S/6$ gives $\tfrac26 \approx 0.333$.Steps 5 and 6 with $N = 5$.
8. $\operatorname{MSE}(cS) = \sigma^4\big[(c(N - 1) - 1)^2 + 2c^2(N - 1)\big]$, minimised at $c^* = \dfrac1{N + 1}$ with $\operatorname{MSE} = \dfrac{2\sigma^4}{N + 1}$; for $N = 5$: $S/(N - 1)$ scores $0.5\sigma^4$, $S/N$ scores $0.36\sigma^4$, $S/(N + 1)$ scores $0.333\sigma^4$The unbiased estimator has the largest error of the three: its lack of bias is bought with variance, and the maximum-likelihood $1/N$ already does better; the best divisor is $N + 1$, which nobody uses because the gain over $1/N$ is small and the argument requires Gaussian data. The MLE page's Problem 5 explains the bias of $S/N$; this problem shows that bias was not the thing to minimise. The check evaluates $\mathbb{E}[S]$ and $\mathbb{E}[S^2]$ exactly by quadrature for small $N$, so the chi-square facts are verified rather than assumed.