Guides / Probability

Bayes' rule: flip the conditional, and divide by the total probability

P(A | B), the probability of A given B, is the chance of A once you know B happened. Bayes' rule turns one conditional into the other: from how likely the evidence is if a hypothesis is true, to how likely the hypothesis is now that you've seen the evidence. The rule is one line. Nearly every wrong answer comes from reading P(A | B) as P(B | A), or from leaving out the denominator that accounts for every other way the evidence could have happened.

Why it works

Conditioning on B throws away every outcome where B didn't happen and rescales what's left so it adds to 1. That's why P(A | B) divides the overlap P(A and B) by P(B). Multiplying back gives the product rule, and the overlap can be written two ways: P(A and B) = P(A | B)P(B) = P(B | A)P(A). Set those equal and divide by P(B), and you have Bayes' rule.

The two conditionals share a numerator and divide it by different things, so they can be far apart. When the hypothesis is rare, most of the evidence comes from the much larger group where it's false, and the denominator P(E) is where that group gets counted.

Example 1: two conditionals from one table

Of 200 customers, 60 canceled. Of those 60, 45 had contacted support; of the 140 who stayed, 35 had. Find P(cancel | support) and P(support | cancel).

1. support: 45 + 35 = 80 customersAdd the two groups that contacted support.
2. P(cancel | support) = 4580 = 0.5625Restrict to the 80 who contacted support; 45 of them canceled.
3. P(support | cancel) = 4560 = 0.75The same 45, now out of the 60 who canceled. The numerator is shared; the condition decides the denominator.

Example 2: the law of total probability, then Bayes

Machine A makes 60% of the parts and 3% of its parts are defective; machine B makes 40% and 5% of its parts are defective. What fraction of all parts are defective, and if a part is defective, how likely is it from B?

1. P(D) = 0.018 + 0.020 = 0.0380.6 · 0.03 + 0.4 · 0.05: every part comes from A or B, so add the defective share from each.
2. P(B | D) = 0.0200.038Bayes' rule, P(D | B)P(B) / P(D): the numerator is B's term from step 1.
3. P(B | D) = 1019 ≈ 0.526B makes 40% of the parts but about 53% of the defects, because its defect rate is higher.

Example 3: a positive test for a rare condition

A condition affects 2% of people. A test for it has sensitivity 95% (P(+ | sick) = 0.95) and specificity 90% (P(− | healthy) = 0.90). Someone tests positive. How likely are they to have the condition?

1. P(+ | healthy) = 1 − 0.90 = 0.10The false positive rate is one minus the specificity.
2. P(+) = 0.019 + 0.098 = 0.1170.95 · 0.02 + 0.10 · 0.98. Positives come from the sick and from the healthy. The second term is about five times the first, because there are 49 healthy people for every sick one.
3. P(sick | +) = 0.0190.117 = 19117Bayes' rule: true positives over all positives.
4. P(sick | +) ≈ 0.162In counts: of 10,000 people, 200 are sick and 190 of them test positive; of the 9,800 healthy, 980 test positive. Only 190 of the 1,170 positives are sick.

Example 4: the same answer in odds, and a second test

Redo Example 3 with odds, then suppose a second, independent run of the same test is also positive.

1. prior odds = 0.020.98 = 149Odds are P(H) / P(not H).
2. LR = 0.950.10 = 9.5The likelihood ratio P(+ | sick) / P(+ | healthy): how much more likely a positive is if sick than if healthy.
3. odds = 9.549 = 1998, P = 19117Posterior odds are prior odds times the ratio. Odds o turn back into a probability as o / (1 + o) = 19 / (19 + 98): the same 0.162 as Example 3.
4. odds = 361196, P ≈ 0.648After the second positive, 19/98 · 9.5 = 361/196, and P = 361/557. Yesterday's posterior is today's prior. Multiplying by the ratio again is only valid if the two results are independent given the person's status; a repeat of a test that fails for the same reason both times is not.

Common mistakes, and the exact line they happen on

1. Reading the sensitivity as the answer

The test from Example 3.

1. P(+ | sick) = 0.95Right so far.
2. so P(sick | +) = 0.95This is the mistake. The two conditionals point in opposite directions. 0.95 is the chance a sick person tests positive; the question is the chance a positive person is sick, which is 19/117 ≈ 0.162. The base rate of 2% is what separates them.

2. Stopping at the numerator

The same test.

1. P(+ | sick)P(sick) = 0.0190.95 · 0.02. Right so far: this is the top of Bayes' rule.
2. P(sick | +) = 0.019This is the mistake. 0.019 is P(sick and +), the chance that a random person is sick and tests positive. To condition on the positive result, divide by P(+) = 0.117, the total over both groups. A quick check: the probabilities of sick and healthy given + must add to 1, and 0.019 + 0.098 = 0.117, not 1, until both are divided by it.

3. Multiplying probabilities that aren't independent

The customers from Example 1.

1. P(cancel) = 0.3, P(support) = 0.460/200 and 80/200. Right so far.
2. P(both) = 0.3 · 0.4 = 0.12This is the mistake. "Both" is cancel and support. P(A and B) = P(A)P(B) only for independent events. Here 45 of the 200 did both, 0.225, almost twice as many: contacting support and canceling go together. In general P(A and B) = P(A | B)P(B) = 0.5625 · 0.4 = 0.225.

Practice

1. P(A) = 0.3, P(B | A) = 0.5 and P(B | not A) = 0.2. Find P(B) and P(A | B).

   Show answer

   P(B) = 0.15 + 0.14 = 0.29, and P(A | B) = 0.150.29 = 1529 ≈ 0.517.

2. 40% of email is spam. The word "winner" appears in 10% of spam and 0.5% of other email. If a message contains "winner", how likely is it to be spam?

   Show answer

   0.4 · 0.10.4 · 0.1 + 0.6 · 0.005 = 0.040.043 = 4043 ≈ 0.930.

3. Two fair coins are flipped. Given that at least one is heads, what is the probability that both are?

   Show answer

   1/3. The outcomes with at least one head are HH, HT and TH, equally likely, and only HH has two.

4. A test is 99% sensitive and 99% specific, and 1 person in 1,000 has the condition. What is P(sick | +)?

   Show answer

   0.99 · 0.0010.99 · 0.001 + 0.01 · 0.999 = 11122 ≈ 0.090. About 9%, even for a 99% test.

5. The prior odds of a hypothesis are 1 to 4, and the evidence is 6 times as likely if it's true as if it's false. What is its probability now?

   Show answer

   Posterior odds 14 · 6 = 32, so the probability is 33 + 2 = 0.6.