Guides / Probability

Maximum likelihood estimation: take the log, set the derivative to zero, then check it's a maximum

Maximum likelihood picks the parameter that makes the data you actually saw most probable. Write the probability of the whole sample as a function of the parameter, take its log so the product becomes a sum, set the derivative to zero, and check that the point you found is a maximum. Most of the losses used to train models, from least squares to cross-entropy, are negative log-likelihoods. The mistakes come from a variance divided by N − 1 and called the MLE, from a distribution written for the wrong count, and from averaging measurements that aren't equally reliable.

Why it works

Once the data are fixed, L(θ) scores each candidate value of the parameter by how probable it makes those data. Independence means the probability of the whole sample is the product of the individual probabilities. The log is strictly increasing, so L and log L peak at the same θ, and the log turns the product into a sum, which can be differentiated term by term. A zero derivative only finds a flat point; a negative second derivative is what makes it a peak.

Minimizing −ℓ(θ) is the same problem as maximizing ℓ(θ), and that is how it shows up in machine learning. With Gaussian noise, −ℓ is the sum of squared errors plus a constant, so least squares is maximum likelihood. With Bernoulli or categorical labels, −ℓ is the cross-entropy loss.

Example 1: the rate of an exponential wait

Four waits between arrivals are 2, 5, 1 and 4 minutes. Model them as exponential with rate λ, density λe^−λx, and find λ̂.

1. ℓ(λ) = Σ (log λ − λx_n) = 4 log λ − 12λThe log of each density is log λ − λx, and the waits add up to 12.
2. ℓ′(λ) = 4/λ − 12 = 0 ⇒ λ̂ = 13In general N/λ − Σx = 0, so λ̂ = N/Σx = 1/x̄.
3. ℓ″(λ̂) = −4/λ̂² = −36 < 0A maximum. λ̂ = 1/3 arrival per minute is one every 3 minutes, the average wait.

Example 2: a geometric count of trials

A player keeps shooting until the first basket. In four rounds it takes 3, 1, 4 and 2 shots. With success probability p per shot, P(X = x) = (1 − p)^{x − 1}p. Find p̂.

1. ℓ(p) = 4 log p + (10 − 4) log(1 − p)Each round contributes log p for the basket and (x − 1) log(1 − p) for the misses. There are 10 shots in all, 4 of them baskets.
2. ℓ′(p) = 4/p − 6/(1 − p) = 0The chain rule on log(1 − p) gives the minus sign.
3. 4(1 − p) = 6p ⇒ p̂ = 0.4Multiply through by p(1 − p).
4. p̂ = 4/10 = 1/x̄Baskets over shots, which is what you'd guess. The mean number of shots is x̄ = 2.5, and 1/2.5 = 0.4.

Example 3: the mean and variance of a normal sample

Find the MLEs of μ and σ² for the data 2, 4, 4, 4, 5, 5, 7, 9, modeled as N(μ, σ²).

1. ℓ(μ, σ²) = −(N/2) log(2πσ²) − Σ(x_n − μ)²/(2σ²)The log of the normal density, summed over the 8 values.
2. ∂ℓ/∂μ = 0 ⇒ μ̂ = x̄ = 40/8 = 5Only the squared errors depend on μ, and their sum is smallest at the mean.
3. Σ(x_n − 5)² = 9 + 1 + 1 + 1 + 0 + 0 + 4 + 16 = 32Squared deviations from μ̂.
4. σ̂² = 32/8 = 4, σ̂ = 2Setting ∂ℓ/∂σ² = 0 gives the average squared deviation, divided by N. The MLE of σ is the square root of the MLE of σ².

Example 4: two measurements with different noise

One instrument reads 10 with noise standard deviation 1; a second reads 13 with standard deviation 2. Both measure the same μ with normal errors. Find μ̂.

1. ℓ(μ) = −(10 − μ)²/2 − (13 − μ)²/8 + constEach squared error is divided by twice its own variance: 2 · 1 and 2 · 4.
2. ℓ′(μ) = (10 − μ) + (13 − μ)/4 = 0Differentiate each square; the 2s cancel.
3. 40 − 4μ + 13 − μ = 0 ⇒ μ̂ = 53/5 = 10.6Multiply by 4 and collect.
4. μ̂ = (1 · 10 + ¼ · 13)/(1 + ¼) = 10.6An average weighted by 1/σ², the precision. The noisier reading counts a quarter as much. The combined precision is 1.25, so the standard error of μ̂ is 1/√1.25 ≈ 0.894, better than either instrument alone.

Common mistakes, and the exact line they happen on

1. Dividing by N − 1 and calling it the MLE

The variance in Example 3.

1. Σ(x_n − 5)² = 32Right so far.
2. σ̂² = 32/7 ≈ 4.571This is the mistake. 32/7 is the unbiased sample variance, what np.var(x, ddof=1) returns. It is a fine estimate, but it isn't the maximum of the likelihood: ∂ℓ/∂σ² = 0 gives 32/8 = 4. The MLE is biased low by the factor (N − 1)/N, which matters for small N.

2. The geometric written as a count of failures

The shooting data of Example 2.

1. 4 rounds, 10 shots, 4 basketsRight so far: each round ends with its basket.
2. ℓ(p) = 4 log p + 10 log(1 − p)This is the mistake. It comes from P(X = x) = (1 − p)^xp and leads to p̂ = 4/14 ≈ 0.286. (1 − p)^xp is the probability of x misses before the basket, but the data count shots, including the basket. It treats 10 shots as 10 misses. With the exponent x − 1 there are 6 misses, and p̂ = 4/10 = 0.4. Check which version of the distribution the data follow before writing ℓ.

3. Averaging readings that aren't equally reliable

The two instruments of Example 4.

1. readings 10 (σ = 1) and 13 (σ = 2)Right so far.
2. μ̂ = (10 + 13)/2 = 11.5This is the mistake. The plain mean is the MLE only when every reading has the same variance. Here the likelihood divides each squared error by its own variance, and its peak is at 10.6. The plain mean has variance (1 + 4)/4 = 1.25, worse than the first instrument's variance of 1 on its own.

Practice

1. Waits of 0.5, 1.5, 2 and 4 seconds are modeled as exponential. Find λ̂.

   Show answer

   The waits add up to 8, so x̄ = 2 and λ̂ = 1/x̄ = 0.5 per second.

2. A coin shows 18 heads in 60 flips. Find the MLE of the probability of heads.

   Show answer

   p̂ = 18/60 = 0.3.

3. Daily counts 3, 0, 2, 5, 0 are modeled as Poisson. Find λ̂.

   Show answer

   λ̂ = x̄ = 10/5 = 2.

4. Find the MLEs of μ and σ² for 1.2, 0.8, 1.6, 0.4, modeled as normal.

   Show answer

   μ̂ = 4/4 = 1. The deviations are 0.2, −0.2, 0.6, −0.6, with squares adding to 0.8, so σ̂² = 0.8/4 = 0.2.

5. For the waits in Problem 1, find the MLE of the mean wait and of P(X > 1) = e^−λ.

   Show answer

   Plug in λ̂ = 0.5: the mean wait is 1/λ̂ = 2 seconds and P(X > 1) = e^−0.5 ≈ 0.607. The MLE of a function of λ is that function of λ̂.