Guides / Probability

Hypothesis tests and p-values: standardize under the null, choose the tails first, and never read p as the chance the null is true

A hypothesis test asks one question: if nothing interesting were going on, how surprising would these data be? The “nothing interesting” is the null hypothesis H₀. The test turns the data into a statistic whose distribution under H₀ is known, usually a standard normal or a t distribution, and the p-value is the probability of a statistic at least as extreme as the one observed, computed as if H₀ were true. Most wrong answers come from three places: the wrong standard error, the wrong number of tails, and reading the p-value as something it is not.

Why it works

If H₀ is true, the central limit theorem says an average or a proportion is roughly normal, centered on the null value, with spread equal to its standard error. Dividing the distance from the null value by that standard error puts every problem on the same scale: z counts how many standard errors the estimate landed from where H₀ says it should be. Values beyond about 2 happen only about 5% of the time by chance, which is where the familiar cutoff 1.96 comes from.

The rule “reject when p ≤ α” then has a guarantee built in: when H₀ really is true, it rejects with probability α. That is all it promises. It says nothing about how likely H₀ is, which depends on how often the effects you test are real, and nothing about how big the effect is, which is what a confidence interval is for.

Example 1: a one-sample z test

A test is scaled to have mean 100 and standard deviation 15. A class of 100 students averages 103.3. Test H₀: μ = 100 against μ ≠ 100 at the 5% level.

1. SE = 15/√100 = 1.5The standard error of an average of 100 scores.
2. z = (103.3 − 100)/1.5Distance from the null value, in standard errors.
3. z = 2.2The class average is 2.2 standard errors above 100.
4. 1 − Φ(2.2) ≈ 0.0139The upper tail beyond 2.2.
5. p = 2 · 0.0139 ≈ 0.0278Two-sided, so both tails count: an average of 96.7 would have been just as surprising. Since 0.0278 < 0.05, reject H₀.

Example 2: a test for a proportion

A classifier is advertised as 90% accurate. On 400 fresh examples it gets 348 right. Is that consistent with 90%, two-sided at 5%?

1. p̂ = 348/400 = 0.87The observed accuracy.
2. SE = √(0.9 · 0.1/400)Under H₀ the accuracy is 0.9, so the standard error uses 0.9, not 0.87.
3. SE = 0.3/20 = 0.015√0.09 = 0.3 and √400 = 20.
4. z = (0.87 − 0.9)/0.015 = −2Two standard errors below the advertised rate.
5. p = 2(1 − Φ(2)) ≈ 0.0455Just under 0.05: reject the claim of 90% at the 5% level. At the 1% level it would stand.

Example 3: a paired t test

Six students take a quiz before and after a review session. Before: 70, 64, 80, 75, 68, 72. After: 73, 69, 79, 79, 70, 77. Did the session change the scores? Test two-sided at 5%.

1. d = 3, 5, −1, 4, 2, 5Pair each student’s two scores: after minus before.
2. d̄ = 18/6 = 3The mean change.
3. s² = 26/5 = 5.2The deviations from 3 are 0, 2, −4, 1, −1, 2, whose squares add up to 26. Divide by n − 1 = 5.
4. SE = √5.2/√6 ≈ 0.931The estimated standard error of d̄.
5. t = 3/0.931 ≈ 3.22The statistic, with 5 degrees of freedom.
6. p ≈ 0.023The cutoff for 5 degrees of freedom is 2.571, and 3.22 > 2.571: reject. The scores went up by about 3 points.

Example 4: what fraction of significant results are real?

A team tests 1000 ideas. Only 10% of them really work, and the tests have 50% power: a real effect comes out significant half the time. Each test uses α = 0.05. Among the significant results, what fraction are false alarms?

1. 900 nulls, 100 real10% of 1000 ideas work.
2. 900 · 0.05 = 45False alarms: each true null passes with probability α.
3. 100 · 0.5 = 50True discoveries, at 50% power.
4. 45/95 ≈ 0.47Almost half of the significant results are false alarms, even though every single test had a 5% false-positive rate. This is Bayes’ rule, and it is why a p-value alone cannot tell you how likely H₀ is.

Common mistakes, and the exact line they happen on

1. Reading the p-value as the chance the null is true

The class average of Example 1.

1. p ≈ 0.0278Right so far.
2. Pr(H₀) ≈ 2.8%This is the mistake. The p-value is computed assuming H₀ is true: it is Pr(data this extreme | H₀). Turning it into Pr(H₀ | data) takes Bayes’ rule, a prior and the power, and Example 4 shows the answer can be nearly 50% when the p-values all look small.

2. Reporting one tail for a two-sided test

Example 1 again.

1. 1 − Φ(2.2) ≈ 0.0139Right so far: the upper tail.
2. p ≈ 0.0139This is the mistake. The alternative was μ ≠ 100, so a result 2.2 standard errors below 100 would have counted too. The two-sided p-value is twice the tail, about 0.0278. A one-sided test is fine only when the direction was fixed before the data were seen.

3. Ignoring the pairing

The before-and-after scores of Example 3, treated as two separate groups.

1. means 71.5 and 74.5Right so far: the difference is still 3.
2. SE = √(31.1/6 + 19.9/6)This is the mistake. These variances, 31.1 and 19.9, measure how much the students differ from one another, which has nothing to do with the session. The standard error comes out as about 2.92, so t ≈ 1.03 and p ≈ 0.33: no effect found. Each student is their own control, so the right test uses the six differences, with t ≈ 3.22 and p ≈ 0.023.

Practice

1. A two-sided z test gives z = 1.5. Find the p-value. Is it significant at 5%?

   Show answer

   p = 2(1 − Φ(1.5)) ≈ 0.134. Not significant.

2. A 95% confidence interval for μ is [4.2, 6.8]. Does a two-sided 5% test reject μ₀ = 4? μ₀ = 5?

   Show answer

   It rejects 4, which is outside the interval, and keeps 5, which is inside.

3. Twenty-five measurements have x̄ = 52 and s = 5. Test μ = 50 two-sided at 5%.

   Show answer

   t = 2/(5/5) = 2 on 24 degrees of freedom. The cutoff is about 2.064, so do not reject; p ≈ 0.057.

4. Ten independent tests are run at α = 0.05, and every null is true. What is the chance that at least one comes out significant?

   Show answer

   1 − 0.95¹⁰ ≈ 0.401.

5. What is the two-sided cutoff for |z| at the 1% level?

   Show answer

   About 2.576, since Φ(2.576) ≈ 0.995.