Practice / Probability for ML

Bayesian inference: conjugate priors, posterior means and MAP estimates

Ten problems on Bayesian inference for a continuous parameter: a Beta posterior from a uniform prior and its MAP estimate, the posterior mean as a weighted average of the prior mean and the data, how the posterior narrows, updating one batch at a time, a Gamma prior on a Poisson error rate, the mean of a normal with known variance and its 95% credible interval, MAP regression with a Gaussian prior as ridge regression, the chance of the next two successes, a Beta prior built from a mean and a standard deviation, and add-one smoothing as a Dirichlet posterior mean, with worked solutions and the mistakes of squaring the posterior mean, forgetting the N in the data's precision, swapping the Gamma updates, inverting the ridge penalty and counting the first batch twice.

Before you start

Maximum likelihood gives one number for a parameter. Bayesian inference gives a distribution: start with a prior density for the parameter, multiply by the likelihood of the data, and normalize. For a few prior and likelihood pairs, called conjugate, the posterior is in the same family as the prior, and the update is just arithmetic on the parameters. These ten problems work through the three conjugate pairs that matter most in practice, the Beta for a probability, the Gamma for a Poisson rate and the normal for a mean, then connect the posterior to the estimates that machine learning actually uses: the MAP estimate, which turns a Gaussian prior into ridge regression, and the posterior mean, which turns a uniform prior into add-one smoothing.

Bayes' rule for a parameter. $p(\theta \mid \text{data}) \propto p(\text{data} \mid \theta)\,p(\theta)$ posterior is proportional to likelihood times prior. The missing constant makes the posterior integrate to $1$ and it does not depend on $\theta$
Beta. $\operatorname{Beta}(\alpha, \beta)$ on $0 < \theta < 1$ has density proportional to $\theta^{\alpha - 1}(1 - \theta)^{\beta - 1}$ Mean $\dfrac{\alpha}{\alpha + \beta}$ mode $\dfrac{\alpha - 1}{\alpha + \beta - 2}$ for $\alpha, \beta > 1$ and variance $\dfrac{\alpha\beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}$ $\operatorname{Beta}(1, 1)$ is the uniform prior.
Beta and binomial. With prior $\operatorname{Beta}(a, b)$ and $k$ successes in $N$ trials, the posterior is $\operatorname{Beta}(a + k, b + N - k)$
Gamma and Poisson. $\operatorname{Gamma}(\alpha, \beta)$ with shape $\alpha$ and rate $\beta$ has density proportional to $\lambda^{\alpha - 1}e^{-\beta\lambda}$ on $\lambda > 0$ mean $\alpha/\beta$ mode $(\alpha - 1)/\beta$ for $\alpha \ge 1$ and variance $\alpha/\beta^2$ With $N$ Poisson counts adding up to $S$ the posterior is $\operatorname{Gamma}(\alpha + S, \beta + N)$
Normal mean, known variance. With prior $\mu \sim \mathcal N(\mu_0, \tau_0^2)$ and $N$ observations from $\mathcal N(\mu, \sigma^2)$ with mean $\bar x$ precisions add: the posterior precision is $1/\tau_0^2 + N/\sigma^2$ and the posterior mean is the precision-weighted average of $\mu_0$ and $\bar x$
Estimates. The MAP estimate is the posterior mode, the maximizer of $\log p(\text{data} \mid \theta) + \log p(\theta)$ With a uniform prior it is the maximum-likelihood estimate. The posterior mean minimizes the expected squared error.
Dirichlet. $\operatorname{Dirichlet}(\alpha_1, \dots, \alpha_K)$ is the Beta for $K$ categories, with mean $\alpha_j/\sum_i \alpha_i$ With category counts $n_j$ the posterior is $\operatorname{Dirichlet}(\alpha_1 + n_1, \dots, \alpha_K + n_K)$

Builds on: Bayes' rule and conditional probability, Maximum likelihood estimation: derivations by hand

Problems

·
A new feature is shown to $12$ users and $3$ of them use it. With a uniform prior on the adoption rate $\theta$ find the posterior, its mean and its MAP estimate, and compare them with the maximum-likelihood estimate.
·
A click-through rate has prior $\operatorname{Beta}(6, 14)$ In $30$ impressions there are $18$ clicks. Find the posterior mean, and show it is a weighted average of the prior mean and the observed rate. What is the weight on the prior?
··
Find the standard deviation of the prior $\operatorname{Beta}(6, 14)$ and of the posterior $\operatorname{Beta}(24, 26)$ from Problem 2.
··
A conversion rate has prior $\operatorname{Beta}(2, 2)$ On day 1, $4$ of $10$ visitors convert; on day 2, $7$ of $10$ Update one day at a time, then all at once, and compare. Give the final posterior mean and MAP estimate.
··
A service's errors per hour are Poisson with rate $\lambda$ and the prior is $\operatorname{Gamma}(3, 2)$ (shape $3$ rate $2$ Over $5$ hours the counts are $3, 1, 4, 0, 2$ Find the posterior, its mean, mode and standard deviation, and compare with the MLE.
··
A sensor's true offset $\mu$ has prior $\mathcal N(50, 16)$ Four readings, each $\mathcal N(\mu, 36)$ have mean $\bar x = 56$ Find the posterior mean and standard deviation of $\mu$ and a $95\%$ credible interval.
···
Take $y = Xw + \varepsilon$ with $\varepsilon \sim \mathcal N(0, \sigma^2 I)$ and prior $w \sim \mathcal N(0, \tau^2 I)$ Show that the MAP estimate is ridge regression with $\lambda = \sigma^2/\tau^2$ Then compute it for $\sigma^2 = 1$ $\tau^2 = \tfrac12$ and
$X = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}, \qquad y = \begin{bmatrix} 1 \\ 2 \\ 4 \end{bmatrix}.$
··
With the posterior $\operatorname{Beta}(4, 10)$ of Problem 1, find the probability that the next user adopts the feature, and the probability that the next two both do.
··
An expert says a defect rate is "about $0.2$ give or take $0.1$ ". Find the Beta prior with mean $0.2$ and standard deviation $0.1$ then update it with $5$ defects in $10$ items and give the posterior mean.
···
A tiny language model sees the words cat, dog and eel $5$ $0$ and $3$ times. Find the posterior mean of the word probabilities under a uniform $\operatorname{Dirichlet}(1, 1, 1)$ prior, and compare it with the maximum-likelihood estimate.

Answers

Posterior $\operatorname{Beta}(4, 10)$ mean $\tfrac27 \approx 0.286$ MAP $0.25$ which equals the MLE $k/N = 0.25$
Posterior mean $0.48 = 0.4 \cdot 0.3 + 0.6 \cdot 0.6$ with weight $w = \tfrac{20}{50} = 0.4$ on the prior
Prior sd $0.1$ posterior sd $\approx 0.070$
Both routes give $\operatorname{Beta}(13, 11)$ with mean $\approx 0.542$ and MAP $\approx 0.545$
Posterior $\operatorname{Gamma}(13, 7)$ mean $\approx 1.857$ mode $\approx 1.714$ sd $\sqrt{13}/7 \approx 0.515$ the MLE is $S/N = 2$
Posterior $\mathcal N(53.84, 5.76)$ sd $2.4$ $95\%$ credible interval $53.84 \pm 1.96 \cdot 2.4 \approx (49.14, 58.54)$
$w_{\text{MAP}} = (\tfrac{14}{15}, \tfrac{19}{15}) \approx (0.933, 1.267)$ ridge with $\lambda = \sigma^2/\tau^2 = 2$
$P(\text{next}) = \tfrac27 \approx 0.286$ $P(\text{next two}) = \tfrac{2}{21} \approx 0.095$
Prior $\operatorname{Beta}(3, 12)$ posterior $\operatorname{Beta}(8, 17)$ with mean $\tfrac{8}{25} = 0.32$
Posterior mean $(\tfrac{6}{11}, \tfrac{1}{11}, \tfrac{4}{11}) \approx (0.545, 0.091, 0.364)$ the MLE gives dog probability $0$

Worked solutions

Problem 1

A new feature is shown to $12$ users and $3$ of them use it. With a uniform prior on the adoption rate $\theta$ find the posterior, its mean and its MAP estimate, and compare them with the maximum-likelihood estimate.

Prior $\operatorname{Beta}(1, 1)$ $k = 3$ $N = 12$ The uniform density on $(0, 1)$ is $\theta^0(1 - \theta)^0$
Posterior $\operatorname{Beta}(1 + 3, 1 + 9) = \operatorname{Beta}(4, 10)$ Add the successes to $\alpha$ and the failures to $\beta$
Mean $\tfrac{4}{14} = \tfrac27 \approx 0.286$ $\alpha/(\alpha + \beta)$
MAP $\tfrac{4 - 1}{14 - 2} = \tfrac{3}{12} = 0.25$ The mode $(\alpha - 1)/(\alpha + \beta - 2)$
Posterior $\operatorname{Beta}(4, 10)$ mean $\tfrac27 \approx 0.286$ MAP $0.25$ which equals the MLE $k/N = 0.25$ A flat prior adds nothing to the log-likelihood, so the posterior peaks where the likelihood does. The mean sits higher because the posterior is skewed to the right.

Problem 2

A click-through rate has prior $\operatorname{Beta}(6, 14)$ In $30$ impressions there are $18$ clicks. Find the posterior mean, and show it is a weighted average of the prior mean and the observed rate. What is the weight on the prior?

Posterior $\operatorname{Beta}(6 + 18, 14 + 12) = \operatorname{Beta}(24, 26)$ $12$ impressions were not clicked.
Posterior mean $\tfrac{24}{50} = 0.48$ $\alpha/(\alpha + \beta)$
Prior mean $\tfrac{6}{20} = 0.3$ observed rate $\tfrac{18}{30} = 0.6$ The two things being averaged.
$\dfrac{a + k}{a + b + N} = w\,\dfrac{a}{a + b} + (1 - w)\,\dfrac{k}{N}$ Holds with $w = \dfrac{a + b}{a + b + N}$ the two terms are $\dfrac{a}{a + b + N}$ and $\dfrac{k}{a + b + N}$
Posterior mean $0.48 = 0.4 \cdot 0.3 + 0.6 \cdot 0.6$ with weight $w = \tfrac{20}{50} = 0.4$ on the priorThe prior counts as $a + b = 20$ pseudo-observations against $30$ real ones. As $N$ grows, $w \to 0$ and the data take over.

Problem 3

Find the standard deviation of the prior $\operatorname{Beta}(6, 14)$ and of the posterior $\operatorname{Beta}(24, 26)$ from Problem 2.

Prior variance $\dfrac{6 \cdot 14}{20^2 \cdot 21} = \dfrac{84}{8400} = 0.01$ $\alpha\beta/((\alpha + \beta)^2(\alpha + \beta + 1))$
Prior standard deviation $0.1$ The square root.
Posterior variance $\dfrac{24 \cdot 26}{50^2 \cdot 51} = \dfrac{624}{127{,}500}$ The same formula with $\alpha + \beta = 50$
$\approx 0.004894$ Less than half the prior variance.
Prior sd $0.1$ posterior sd $\approx 0.070$ The variance shrinks roughly like $1/(\alpha + \beta)$ the prior's $20$ pseudo-observations plus the $30$ real ones.

Problem 4

A conversion rate has prior $\operatorname{Beta}(2, 2)$ On day 1, $4$ of $10$ visitors convert; on day 2, $7$ of $10$ Update one day at a time, then all at once, and compare. Give the final posterior mean and MAP estimate.

After day 1: $\operatorname{Beta}(2 + 4, 2 + 6) = \operatorname{Beta}(6, 8)$ $6$ visitors did not convert.
After day 2: $\operatorname{Beta}(6 + 7, 8 + 3) = \operatorname{Beta}(13, 11)$ Day 1's posterior is day 2's prior.
All at once: $11$ of $20$ so $\operatorname{Beta}(2 + 11, 2 + 9)$ The same $\operatorname{Beta}(13, 11)$
Mean $\tfrac{13}{24} \approx 0.542$ MAP $\tfrac{12}{22} \approx 0.545$ $\alpha/(\alpha + \beta)$ and $(\alpha - 1)/(\alpha + \beta - 2)$
Both routes give $\operatorname{Beta}(13, 11)$ with mean $\approx 0.542$ and MAP $\approx 0.545$ Multiplying the likelihoods of two independent batches is the same as multiplying by one and then the other. Each observation must be used exactly once (Mistake 5).

Problem 5

A service's errors per hour are Poisson with rate $\lambda$ and the prior is $\operatorname{Gamma}(3, 2)$ (shape $3$ rate $2$ Over $5$ hours the counts are $3, 1, 4, 0, 2$ Find the posterior, its mean, mode and standard deviation, and compare with the MLE.

$S = 3 + 1 + 4 + 0 + 2 = 10$ errors in $N = 5$ hoursOnly the total and the number of hours enter the likelihood $\prod e^{-\lambda}\lambda^{x_i} \propto \lambda^S e^{-N\lambda}$
Posterior $\propto \lambda^{10}e^{-5\lambda} \cdot \lambda^{2}e^{-2\lambda}$ The likelihood times the prior density.
$= \lambda^{12}e^{-7\lambda}$ so $\operatorname{Gamma}(13, 7)$ Shape $\alpha + S$ rate $\beta + N$
Mean $\tfrac{13}{7} \approx 1.857$ mode $\tfrac{12}{7} \approx 1.714$ $\alpha/\beta$ and $(\alpha - 1)/\beta$
Posterior $\operatorname{Gamma}(13, 7)$ mean $\approx 1.857$ mode $\approx 1.714$ sd $\sqrt{13}/7 \approx 0.515$ the MLE is $S/N = 2$ The prior mean is $3/2 = 1.5$ worth $2$ hours of data, so the posterior mean sits between $1.5$ and $2$ closer to the data's $5$ hours.

Problem 6

A sensor's true offset $\mu$ has prior $\mathcal N(50, 16)$ Four readings, each $\mathcal N(\mu, 36)$ have mean $\bar x = 56$ Find the posterior mean and standard deviation of $\mu$ and a $95\%$ credible interval.

Prior precision $\tfrac{1}{16} = 0.0625$ One over the prior variance.
Data precision $\tfrac{N}{\sigma^2} = \tfrac{4}{36} \approx 0.1111$ The mean of $4$ readings has variance $36/4 = 9$
Posterior precision $0.1736$ variance $5.76$ Precisions add; $1/0.1736 = 5.76$
Mean $0.36 \cdot 50 + 0.64 \cdot 56 = 53.84$ Weights are the precisions over their sum: $0.0625/0.1736 = 0.36$
Posterior $\mathcal N(53.84, 5.76)$ sd $2.4$ $95\%$ credible interval $53.84 \pm 1.96 \cdot 2.4 \approx (49.14, 58.54)$ The posterior sd is smaller than both the prior's $4$ and the data's $3$ The interval is a probability statement about $\mu$ given the data and the prior.

Problem 7

Take $y = Xw + \varepsilon$ with $\varepsilon \sim \mathcal N(0, \sigma^2 I)$ and prior $w \sim \mathcal N(0, \tau^2 I)$ Show that the MAP estimate is ridge regression with $\lambda = \sigma^2/\tau^2$ Then compute it for $\sigma^2 = 1$ $\tau^2 = \tfrac12$ and

X = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{bmatrix}, \qquad y = \begin{bmatrix} 1 \\ 2 \\ 4 \end{bmatrix}.

$-\log p(w \mid y) = \tfrac{1}{2\sigma^2}\lVert y - Xw\rVert^2 + \tfrac{1}{2\tau^2}\lVert w\rVert^2 + c$ The log-likelihood and the log-prior, with the terms that do not depend on $w$ collected in $c$
Multiply by $2\sigma^2$ minimize $\lVert y - Xw\rVert^2 + \lambda\lVert w\rVert^2$ A positive factor does not move the minimizer; $\lambda = \sigma^2/\tau^2$
$w_{\text{MAP}} = (X^\top X + \lambda I)^{-1}X^\top y$ Set the gradient $-2X^\top(y - Xw) + 2\lambda w$ to $0$
$\lambda = 2$ $X^\top X + 2I = \begin{bmatrix} 4 & 1 \\ 1 & 4 \end{bmatrix}$ and $X^\top y = \begin{bmatrix} 5 \\ 6 \end{bmatrix}$ $X^\top X$ is $\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$
$\tfrac{1}{15}\begin{bmatrix} 4 & -1 \\ -1 & 4 \end{bmatrix}\begin{bmatrix} 5 \\ 6 \end{bmatrix} = \tfrac{1}{15}\begin{bmatrix} 14 \\ 19 \end{bmatrix}$ The $2 \times 2$ inverse, with determinant $15$
$w_{\text{MAP}} = (\tfrac{14}{15}, \tfrac{19}{15}) \approx (0.933, 1.267)$ ridge with $\lambda = \sigma^2/\tau^2 = 2$ Least squares gives $(\tfrac43, \tfrac73)$ the prior pulls both weights toward $0$ A tighter prior (smaller $\tau^2$ means a larger $\lambda$

Problem 8

With the posterior $\operatorname{Beta}(4, 10)$ of Problem 1, find the probability that the next user adopts the feature, and the probability that the next two both do.

$P(\text{next}) = E[\theta] = \tfrac{4}{14} = \tfrac27$ Average $P(\text{success} \mid \theta) = \theta$ over the posterior.
$P(\text{next two}) = E[\theta^2]$ Given $\theta$ the two are independent with probability $\theta^2$ average that over the posterior.
$E[\theta^2] = \dfrac{\alpha(\alpha + 1)}{(\alpha + \beta)(\alpha + \beta + 1)}$ The second moment of a Beta.
$= \dfrac{4 \cdot 5}{14 \cdot 15} = \dfrac{20}{210}$ With $\alpha = 4$ $\beta = 10$
$P(\text{next}) = \tfrac27 \approx 0.286$ $P(\text{next two}) = \tfrac{2}{21} \approx 0.095$ Larger than $(\tfrac27)^2 \approx 0.082$ because $E[\theta^2] = (E\theta)^2 + \operatorname{Var}(\theta)$ (Mistake 1).

Problem 9

An expert says a defect rate is "about $0.2$ give or take $0.1$ ". Find the Beta prior with mean $0.2$ and standard deviation $0.1$ then update it with $5$ defects in $10$ items and give the posterior mean.

With $m = \tfrac{\alpha}{\alpha + \beta}$ and $n = \alpha + \beta$ $\operatorname{Var} = \tfrac{m(1 - m)}{n + 1}$ The Beta variance rewritten in terms of the mean and the total $n$
$\tfrac{0.2 \cdot 0.8}{n + 1} = 0.01$ so $n + 1 = 16$ $0.1^2 = 0.01$
$n = 15$ $\alpha = 0.2 \cdot 15 = 3$ $\beta = 12$ $\alpha = mn$ $\beta = (1 - m)n$
Posterior $\operatorname{Beta}(3 + 5, 12 + 5) = \operatorname{Beta}(8, 17)$ $5$ defects and $5$ good items.
Prior $\operatorname{Beta}(3, 12)$ posterior $\operatorname{Beta}(8, 17)$ with mean $\tfrac{8}{25} = 0.32$ The expert's opinion is worth $15$ items, so $10$ items at a rate of $0.5$ move the mean from $0.2$ only to $0.32$

Problem 10

A tiny language model sees the words cat, dog and eel $5$ $0$ and $3$ times. Find the posterior mean of the word probabilities under a uniform $\operatorname{Dirichlet}(1, 1, 1)$ prior, and compare it with the maximum-likelihood estimate.

Posterior $\operatorname{Dirichlet}(1 + 5, 1 + 0, 1 + 3) = \operatorname{Dirichlet}(6, 1, 4)$ Add each count to its parameter.
Posterior mean $\tfrac{1}{11}(6, 1, 4)$ Each parameter over their sum $11$
MLE $\tfrac18(5, 0, 3) = (0.625, 0, 0.375)$ Counts over the total $8$
$\dfrac{n_j + 1}{N + K}$ with $N = 8$ $K = 3$ The posterior mean adds one to every count and $K$ to the total.
Posterior mean $(\tfrac{6}{11}, \tfrac{1}{11}, \tfrac{4}{11}) \approx (0.545, 0.091, 0.364)$ the MLE gives dog probability $0$ This is add-one (Laplace) smoothing. Under the MLE an unseen word has probability $0$ so any sentence containing it has likelihood $0$ and log-likelihood $-\infty$

Where this goes wrong

1. Squaring the posterior mean for two future successes

Problem 8 asks for two adoptions in a row, and two independent events multiply.

$P(\text{next}) = E[\theta] = \tfrac27$ Right so far.
“The next two are independent, so square it.”They are independent only given $\theta$ Unconditionally, a first adoption is evidence that $\theta$ is high, which raises the chance of the second.
$P(\text{next two}) = (\tfrac27)^2 = \tfrac{4}{49} \approx 0.082$ The right value is $E[\theta^2] = \tfrac{2}{21} \approx 0.095$ The gap is $\operatorname{Var}(\theta) = \tfrac{2}{21} - \tfrac{4}{49} \approx 0.014$ the posterior's own uncertainty.

2. Forgetting the N in the data's precision

Problem 6 has a prior variance of $16$ and a reading variance of $36$ and the two variances are the numbers on the page.

Prior precision $\tfrac{1}{16}$ Right so far.
“The data precision is $\tfrac{1}{36}$ ”That is the precision of a single reading. The mean of $4$ readings has variance $36/4 = 9$ so the data precision is $4/36$
Posterior variance $1/(\tfrac{1}{16} + \tfrac{1}{36}) \approx 11.08$ mean $\approx 51.85$ The right answer is variance $5.76$ and mean $53.84$ Dropping $N$ treats four readings as one, so the posterior stays too close to the prior and too wide (sd $\approx 3.33$ instead of $2.4$

3. Swapping the Gamma updates

The Poisson update adds two numbers to the two Gamma parameters, and it is easy to pair them the wrong way.

Prior $\operatorname{Gamma}(3, 2)$ $S = 10$ errors in $N = 5$ hoursRight so far: Problem 5.
“Add the hours to the shape and the errors to the rate.”The likelihood is $\lambda^S e^{-N\lambda}$ the count $S$ is a power of $\lambda$ which belongs with the shape, and the exposure $N$ multiplies $\lambda$ in the exponent, which belongs with the rate.
Posterior $\operatorname{Gamma}(3 + 5, 2 + 10) = \operatorname{Gamma}(8, 12)$ mean $\approx 0.667$ The right posterior is $\operatorname{Gamma}(13, 7)$ with mean $\approx 1.857$ A rate of $0.667$ errors an hour is below both the prior mean $1.5$ and the observed $2$ which a posterior mean can never be.

4. Inverting the ridge penalty

Problem 7's prior variance is $\tau^2 = \tfrac12$ and the noise variance is $\sigma^2 = 1$ and the ratio is easy to write upside down.

Minimize $\tfrac{1}{2\sigma^2}\lVert y - Xw\rVert^2 + \tfrac{1}{2\tau^2}\lVert w\rVert^2$ Right so far.
“ $\lambda = \tau^2/\sigma^2$ ”Multiplying through by $2\sigma^2$ leaves $\sigma^2/\tau^2$ in front of $\lVert w\rVert^2$ A tight prior, small $\tau^2$ has to mean a strong penalty.
$\lambda = \tfrac12$ so $w \approx (1.238, 1.905)$ With $\lambda = 2$ the answer is $(0.933, 1.267)$ The inverted ratio makes a confident prior shrink the weights less than a vague one would.

5. Counting the first batch twice

In Problem 4, day 1's posterior becomes day 2's prior, and then all the data so far are used for the update.

After day 1: $\operatorname{Beta}(6, 8)$ Right so far.
“Update with everything seen: $11$ of $20$ ”Day 1's $4$ of $10$ is already inside $\operatorname{Beta}(6, 8)$ The update should use only the new data, $7$ of $10$
Posterior $\operatorname{Beta}(6 + 11, 8 + 9) = \operatorname{Beta}(17, 17)$ mean $0.5$ The right posterior is $\operatorname{Beta}(13, 11)$ mean $\approx 0.542$ Counting data twice also makes the posterior too confident: sd $\approx 0.085$ instead of $\approx 0.100$

Print this set: bayesian-inference-and-conjugate-priors.pdf (problems, answers, and worked solutions on separate pages).