EulerPad

Guides / Probability

Conditional expectation and the law of total variance: average inside each slice, then add the spread between the slices

Conditional expectation answers “what is the average of Y once I know X?”. For one value x it is a number, the mean of Y in the slice X = x. Over all values of X at once it is a function of X, and so a random variable with its own mean and variance. Two laws do almost all the work: the tower law, which averages the slice means back to E[Y], and the law of total variance, which splits Var(Y) into a part inside the slices and a part between them. Most wrong answers keep only one of the two parts.

Short answer

One slice: E[Y | X = x] = Σy y Pr(Y = y | X = x), or the integral of y against the conditional density. All slices: writing g(x) = E[Y | X = x], the conditional expectation E[Y | X] is the random variable g(X). Tower law: E[Y] = E[E[Y | X]], each slice mean weighted by how likely its slice is. Total variance: Var(Y) = E[Var(Y | X)] + Var(E[Y | X]), the average spread inside the slices plus the spread of the slice means.

Why it works

Split the outcomes into slices by the value of X. The overall average of Y is a weighted average of the slice averages, with each slice weighted by its probability: that is the tower law, and it is the law of total probability with y multiplied in. Nothing is lost by averaging in two stages, as long as the second stage uses the slice probabilities as weights.

For the variance, measure each value of Y from the overall mean in two hops: from Y to its slice mean, then from the slice mean to the overall mean. Squaring and averaging gives the two squared hops plus a cross term. Inside any slice the first hop averages to zero, because that is what the slice mean is, so the cross term vanishes. What is left is the average squared first hop, E[Var(Y | X)], plus the average squared second hop, Var(E[Y | X]). Both are never negative, so knowing X can only make the remaining spread, the first term, smaller than Var(Y).

Example 1: a die decides how many coins to flip

Roll a fair die and call the result X. Then flip X fair coins and let Y be the number of heads. Find E[Y | X] and E[Y].

  1. Y | X = x ~ Bin(x, 1/2)Once the die shows x, Y counts heads in x independent flips.
  2. E[Y | X = x] = x/2The mean of a binomial is np.
  3. E[Y | X] = X/2The same rule for every slice, written as a function of X. It is random: it is 0.5 if the die shows 1 and 3 if it shows 6.
  4. E[Y] = E[X]/2The tower law: average the slice means over the die.
  5. E[Y] = 3.5/2 = 1.75The mean of a fair die is 3.5. Working out the whole distribution of Y and averaging it gives the same 1.75, with far more arithmetic.

Example 2: the variance of the same count

For the die and coins of Example 1, find Var(Y).

  1. Var(Y | X) = X/4A binomial has variance np(1 − p), here x · 1/2 · 1/2.
  2. E[Var(Y | X)] = 3.5/4 = 7/8The spread inside the slices, averaged over the die.
  3. Var(X) = 91/6 − 3.52 = 35/12E[X2] = (1 + 4 + … + 36)/6 = 91/6.
  4. Var(X/2) = (35/12)/4 = 35/48The spread of the slice means. A constant factor comes out of a variance squared.
  5. Var(Y) = 7/8 + 35/48 = 77/48About 1.604. Slightly more than half of it, 42/48, is coin noise; the rest is not knowing how many coins there will be.

Example 3: two groups with different means

A course has two sections. Section A has 40 students with a mean score of 70 and a standard deviation of 10; section B has 60 students with a mean of 80 and a standard deviation of 5. Find the mean and standard deviation of all 100 scores.

  1. weights 0.4 and 0.6Let X be the section of a student picked at random: the slices are the sections.
  2. E[Y] = 0.4 · 70 + 0.6 · 80 = 76The tower law, with each section weighted by its size.
  3. E[Var(Y | X)] = 40 + 15 = 55The variances are 102 = 100 and 52 = 25, weighted: 0.4 · 100 and 0.6 · 25.
  4. Var(E[Y | X]) = 24The slice means 70 and 80 differ by 10. A variable with two values a and b, taken with probabilities p and 1 − p, has variance p(1 − p)(b − a)2 = 0.24 · 100.
  5. Var(Y) = 79, SD ≈ 8.89The combined scores are more spread out than either section, because the sections sit 10 points apart. Check with second moments: 0.4 · (100 + 4900) + 0.6 · (25 + 6400) = 5855, and 5855 − 762 = 79.

Example 4: a stick broken twice

Break a stick of length 1 at a uniform random point and keep the left piece, of length X ~ U(0, 1). Break that piece at a uniform random point and keep the left part, of length Y. Find E[Y] and Var(Y).

  1. Y | X ~ U(0, X)Given the first piece has length x, the second break is uniform on it.
  2. E[Y | X] = X/2The middle of [0, x].
  3. E[Y] = E[X]/2 = 1/4The tower law.
  4. Var(Y | X) = X2/12A uniform on an interval of length x has variance x2/12.
  5. E[X2/12] = (1/3)/12 = 1/36Average over X, using E[X2] = 1/3 for U(0, 1).
  6. Var(X/2) = (1/12)/4 = 1/48The spread of the slice means: Var(X) = 1/12.
  7. Var(Y) = 1/36 + 1/48 = 7/144About 0.0486. Check: E[Y2 | X] = X2/3, so E[Y2] = 1/9, and 1/9 − 1/16 = 7/144.

Common mistakes, and the exact line they happen on

1. Stopping after the first term

The two sections of Example 3.

  1. E[Var(Y | X)] = 55Right so far.
  2. SD = √55 ≈ 7.42This is the mistake. Averaging the section variances measures only the spread inside the sections. The section means 70 and 80 are also apart, and that adds Var(E[Y | X]) = 24. The right variance is 79, a standard deviation of about 8.89.

2. Not squaring the constant

The die and coins of Example 2.

  1. E[Y | X] = X/2Right so far.
  2. Var(X/2) = (35/12)/2 = 35/24This is the mistake. Halving X halves every distance from the mean, so it divides the squared distances by 4: Var(cX) = c2Var(X). With 35/24 the total comes out as 7/8 + 35/24 = 7/3, instead of the right 77/48 ≈ 1.604.

3. Plugging the mean of X into Var(Y | X)

The broken stick of Example 4.

  1. Var(Y | X) = X2/12Right so far.
  2. E[Var(Y | X)] = 1/48This is the mistake. This puts E[X] = 1/2 into X2/12, giving (1/2)2/12. That is not the same as averaging it, because E[X2] = 1/3, not 1/4. The total would be 1/48 + 1/48 = 1/24, and the right Var(Y) is 7/144. The shortcut is safe only when the expression is a straight line in X, as E[Y | X] = X/2 was in step 3 of Example 4.

Practice

  1. A fair coin picks a die: heads, a six-sided die; tails, a four-sided die numbered 1 to 4. Find the mean of the roll.

    Show answer

    0.5 · 3.5 + 0.5 · 2.5 = 3.

  2. For the same setup, find the variance of the roll.

    Show answer

    The two dice have variances 35/12 and 5/4, which average to 25/12. The means 3.5 and 2.5 add 0.25 · 12 = 1/4. Total 25/12 + 3/12 = 7/3 ≈ 2.33.

  3. Emails arrive at a Poisson rate of 5 an hour, and each takes a mean of 2 minutes to answer with a standard deviation of 1 minute, independently. Find the mean and standard deviation of the minutes spent per hour.

    Show answer

    Mean 5 · 2 = 10 minutes. Variance 5 · 12 + 5 · 22 = 25, so the standard deviation is 5 minutes: the Poisson count has variance 5.

  4. X ~ N(1, 9) and, given X, Y ~ N(X, 4). Find E[Y] and Var(Y).

    Show answer

    E[Y] = E[X] = 1 and Var(Y) = 4 + Var(X) = 13.

  5. If E[Y | X] = 3 whatever the value of X, what are E[Y] and Cov(X, Y)?

    Show answer

    E[Y] = 3, and E[XY] = E[X · E[Y | X]] = 3E[X], so Cov(X, Y) = 0. All of Var(Y) is then inside the slices.

Practice this

Printable PDF: conditional-expectation-and-the-law-of-total-variance.pdf

Related guides

All guides