5.8.3 Hypothesis tests

Null and alternative hypotheses

A coin is tossed 100 times and shows 60 heads. Is it fair? A hypothesis test answers such a question by asking how surprising the data would be if a stated claim were true.

The null hypothesis $H_0$ is the claim under test, usually "nothing special": here $p = 0.5$ , with $p$ the chance of heads.
The alternative hypothesis $H_1$ is what the test looks for instead: here $p \ne 0.5$ . It is two-sided, because a bias either way counts.
The test statistic is a number computed from the data whose distribution is known when $H_0$ is true. Here it is the number of heads $X$ , which is $\operatorname{Bin}(100, 0.5)$ under $H_0$ (lesson 5.2.2).

Under $H_0$ , $X$ has mean 50 and standard deviation $\sqrt{100 \times 0.5 \times 0.5} = 5$ (lesson 5.3.2). Counts far from 50, in either direction, speak against $H_0$ .

The p-value

Rule: p-value

\begin{gathered} p = P(\text{a statistic at least as extreme} \\ \text{as the one observed} \mid H_0) \end{gathered}

Small $p$ : data like these would be rare if $H_0$ were true, so they are evidence against it. The p-value is a probability computed assuming $H_0$ ; it is not the probability that $H_0$ is true.

Sixty heads is 10 away from 50, so "at least as extreme" means $\lvert X - 50 \rvert \ge 10$ . Each tail has probability 0.0284, so $p = 0.0569$ .

If the coin is fair, the highlighted counts are at least as far from 50 as the 60 heads seen. Their total probability, 0.0569, is the p-value.

To decide, fix a significance level $\alpha$ before looking at the data, often 0.05 or 0.01.

Rule: Significance level

\text{reject } H_0 \text{ when } p \le \alpha

The result is then statistically significant at level $\alpha$ . Otherwise the data do not reject $H_0$ , which is not the same as showing it is true.

60 heads: $p = 0.0569$ , above 0.05, so the data do not reject fairness at the 5% level.
61 heads: $p = 0.0352$ , so they do. One toss changes the verdict, which is why a report should give the p-value, not only "reject" or "keep".
Not the chance that $H_0$ is true. The p-value is $P(\text{data this extreme} \mid H_0)$ . The reverse, $P(H_0 \mid \text{data})$ , needs Bayes' rule and a prior (lesson 5.1.4).

A normal approximation, $z = \frac{60 - 50}{5} = 2$ , gives $p = 2N(-2) \approx 0.0455$ instead of 0.0569. Near the cut-off it flips the verdict, so use the exact distribution when you have it.

P(X \ge 60)

import numpy as np from scipy.stats import binom, binomtest, norm for heads in [60, 61]: k = heads - 50 p = binom.sf(50 + k - 1, 100, 0.5) + binom.cdf(50 - k, 100, 0.5) print(heads, round(binom.sf(heads - 1, 100, 0.5), 4), round(p, 4), round(binomtest(heads, 100, 0.5).pvalue, 4)) print(round(2 * norm.sf((60 - 50) / 5), 4)) X = np.random.default_rng(583).binomial(100, 0.5, size=100_000) print(np.mean(np.abs(X - 50) >= 10))

⌘/Ctrl + Enter runs the cell

Two kinds of error

A test can go wrong in two ways.

	$H_0$ true	$H_0$ false
reject $H_0$	error of the first kind (Type I): a false alarm	correct
keep $H_0$	correct	error of the second kind (Type II): a miss

Rule: Error rates and power

\begin{gathered} P(\text{Type I}) = \alpha \\[4pt] P(\text{Type II}) = \beta \\[4pt] \text{power} = 1 - \beta \end{gathered}

$\alpha$ is chosen; $\beta$ (beta) depends on how far the truth is from $H_0$ , on $n$ and on $\alpha$ . The power is the chance of rejecting $H_0$ when it is false.

Take draws from $\mathcal{N}(\mu, 1)$ and test $H_0: \mu = 0$ against $\mu \ne 0$ with $n = 25$ . The statistic $z = \frac{\bar{x}}{1/\sqrt{25}} = 5\bar{x}$ is standard normal under $H_0$ , so rejecting when $\lvert z \rvert > 1.96$ gives $\alpha = 0.05$ .

If in fact $\mu = 0.5$ , then $z \sim \mathcal{N}(2.5, 1)$ . It lands beyond $\pm 1.96$ with probability 0.7054, the power, so $\beta = 0.2946$ .

The same rule judged under two truths. If $\mu = 0$ , rejecting is a false alarm, with probability $\alpha$ . If $\mu = 0.5$ , keeping $H_0$ is a miss, with probability $\beta$ .

Power grows with the sample and with the size of the effect:

	$n = 10$	$n = 25$	$n = 50$	$n = 100$
power when $\mu = 0.5$	0.3526	0.7054	0.9424	0.9988

At $n = 25$ the power is 0.1701 when $\mu = 0.2$ and 0.9988 when $\mu = 1$ . Cutting $\alpha$ to 0.01 (cut-off 2.576) lowers it from 0.7054 to 0.4697: fewer false alarms cost more misses.

How many draws? To detect a difference $\delta$ (delta) with 80% power at $\alpha = 0.05$ , the test needs about $n = \left(\frac{(1.96 + 0.8416)\,\sigma}{\delta}\right)^2$ draws, with $0.8416 = z_{0.8}$ . For $\delta = 0.5$ and $\sigma = 1$ that is 31.4, so 32 draws. Case study 5.8.5 asks the same question about years of returns.

\mathcal{N}(\mu\sqrt{n}, 1)

import numpy as np from scipy.stats import norm def power(mu, n, z=1.96): shift = mu * np.sqrt(n) return norm.sf(z - shift) + norm.cdf(-z - shift) print([round(float(power(0.5, n)), 4) for n in [10, 25, 50, 100]]) print([round(float(power(mu, 25)), 4) for mu in [0.2, 0.5, 1.0]], round(power(0.5, 25, z=2.576), 4)) print(round(norm.ppf(0.8), 4), round(((1.96 + norm.ppf(0.8)) / 0.5) ** 2, 2)) rng = np.random.default_rng(5831) for mu in [0.0, 0.5]: z = rng.normal(mu, 1, size=(100_000, 25)).mean(axis=1) * 5 print(mu, round(np.mean(np.abs(z) > 1.96), 4))

⌘/Ctrl + Enter runs the cell

The t test for a mean

To test a claimed mean $\mu_0$ with $\sigma$ unknown, standardise with $s$ , as lesson 5.8.2 did for the t interval.

Rule: One-sample t test

\begin{gathered} t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \\[4pt] p = 2\,P(T_{n-1} \ge \lvert t \rvert) \end{gathered}

$H_0: \mu = \mu_0$ against $\mu \ne \mu_0$ . Under $H_0$ the statistic has the t distribution with $n - 1$ degrees of freedom, exactly for normal draws and approximately for large samples otherwise. SciPy: ttest_1samp(x, mu0).

Five measurements. Lesson 5.8.2's values 10.2, 9.6, 10.9, 10.4 and 9.9 have $\bar{x} = 10.2$ and standard error 0.2214.

Claimed value 10: $t = \frac{0.2}{0.2214} \approx 0.9035$ and $p = 0.4173$ . No evidence against 10.
Claimed value 9.5: $t \approx 3.1623$ and $p = 0.0341$ , so 9.5 is rejected at the 5% level.

Lesson 5.8.2's 95% interval, 9.5854 to 10.8146, contains 10 and leaves out 9.5. That is no coincidence.

Rule: Tests and intervals

\begin{gathered} \text{reject } \mu_0 \text{ at level } \alpha \\ \iff \mu_0 \text{ is outside the } (1 - \alpha) \text{ interval} \end{gathered}

For the two-sided t test and the t interval from the same data. The interval is the set of claimed values the test does not reject.

t

import numpy as np from scipy.stats import t, ttest_1samp x = np.array([10.2, 9.6, 10.9, 10.4, 9.9]) n = len(x) for mu0 in [10, 9.5]: tstat = (x.mean() - mu0) / (x.std(ddof=1) / np.sqrt(n)) p = 2 * t.sf(abs(tstat), n - 1) res = ttest_1samp(x, mu0) print(mu0, round(tstat, 4), round(p, 4), round(res.statistic, 4), round(res.pvalue, 4)) print(round(t.sf(3.1623, n - 1), 4), round(ttest_1samp(x, 9.5, alternative="greater").pvalue, 4))

⌘/Ctrl + Enter runs the cell

A loaded die. Make a six half again as likely as each other face. Its probability is then 0.2308, and the die's mean is 3.6923, not 3.5. So $H_0: \mu = 3.5$ is false.

Sixty seeded rolls give $\bar{x} \approx 3.4667$ , $t \approx -0.1372$ and $p = 0.8913$ : no evidence at all. That is an error of the second kind.

That is the usual outcome, not bad luck. With 60 rolls the t test detects this die only 13.6% of the time, in a seeded simulation. For 80% power it needs about 666 rolls, by the sample-size formula with $\sigma \approx 1.7709$ .

A test with little power says little when it keeps $H_0$ .

Try it The first lines build the loaded die's probabilities, mean and standard deviation. Then one sample of 60 rolls and its t test. The last lines are the power, as the share of 20,000 seeded samples of 60 with a p-value below 0.05. Then comes the formula's sample size for 80%.

import numpy as np from scipy.stats import norm, ttest_1samp faces = np.arange(1, 7) w = np.array([1, 1, 1, 1, 1, 1.5]) / 6.5 mu = faces @ w sd = np.sqrt(((faces - mu) ** 2) @ w) print(round(w[5], 4), round(mu, 4), round(sd, 4)) rolls = np.random.default_rng(5832).choice(faces, p=w, size=60) res = ttest_1samp(rolls, 3.5) print(round(rolls.mean(), 4), round(res.statistic, 4), round(res.pvalue, 4)) many = np.random.default_rng(5834).choice(faces, p=w, size=(20_000, 60)) print(round(np.mean(ttest_1samp(many, 3.5, axis=1).pvalue < 0.05), 3)) print(round(((1.96 + norm.ppf(0.8)) * sd / (mu - 3.5)) ** 2, 1))

⌘/Ctrl + Enter runs the cell

One-sided tests

Sometimes only one direction matters: a new process should raise the mean, not lower it. Then $H_1: \mu > \mu_0$ , and only large values of $t$ count against $H_0$ .

Rule: One-sided p-value

\begin{gathered} H_1: \mu > \mu_0 \\[4pt] p = P(T_{n-1} \ge t) \end{gathered}

For $H_1: \mu < \mu_0$ use $P(T_{n-1} \le t)$ . In SciPy, alternative="greater" or "less".

For the claimed value 9.5, the one-sided p-value is 0.0171, half of the two-sided 0.0341, because $t$ is positive. Had $t$ been negative, the one-sided p-value would be above 0.5.

Choose the side before seeing the data. Picking it afterwards, to match the sign of $t$ , halves every p-value and doubles the false alarms.

Many tests at once

Each test at $\alpha = 0.05$ raises a false alarm 5% of the time when its $H_0$ is true. Run many and some will.

With 20 independent tests of true nulls, the chance of at least one rejection is $1 - 0.95^{20} \approx 0.6415$ , and one false rejection is expected. In a seeded simulation of 10,000 batches of 20 t tests, 63.46% of batches had at least one. The smallest of the 20 p-values averaged 0.0478.

Rule: Bonferroni correction

\text{test each of } m \text{ hypotheses at } \frac{\alpha}{m}

Then the chance of any false alarm among the $m$ is at most $\alpha$ . For 20 tests, each is run at 0.0025, and the chance of any false rejection is 0.0488. The price is power: each test now needs stronger evidence.

Significant is not the same as large. Take 10,000 draws with standard deviation 1. A sample mean of 0.02 gives $t = 2$ and $p = 0.0455$ : significant, but tiny.

Report the estimate and its interval alongside the p-value. Case study 5.8.5 meets both problems with trading strategies.

H_0: \mu = 0

import numpy as np from scipy.stats import t, ttest_1samp print(round(1 - 0.95**20, 4), 0.05 / 20, round(1 - (1 - 0.05 / 20) ** 20, 4)) data = np.random.default_rng(5833).standard_normal((10_000, 20, 30)) p = ttest_1samp(data, 0, axis=2).pvalue print(round(np.mean((p < 0.05).any(axis=1)), 4), round(np.mean((p < 0.05).sum(axis=1)), 3)) print(round(np.mean((p < 0.0025).any(axis=1)), 4), round(np.mean(p.min(axis=1)), 4)) print(round(0.02 / (1 / np.sqrt(10_000)), 4), round(2 * t.sf(2, 9_999), 4))

⌘/Ctrl + Enter runs the cell

Common mistakes

Reading $p$ as the chance that $H_0$ is true. It is the chance of data this extreme if $H_0$ is true.
Treating "not rejected" as "proved". The loaded die kept $H_0$ with $p = 0.8913$ ; the test had 13.6% power.
Choosing $\alpha$ or the side after seeing the data. Both make false alarms more common than $\alpha$ says.
Equating significant with important. A tiny effect is significant with enough data; give the estimate and its interval.
Ignoring how many tests were run. Twenty tests of true nulls give at least one $p \le 0.05$ about 64% of the time.
Trusting an approximation at the cut-off. For 60 heads the normal approximation gives 0.0455 and the exact test 0.0569.

Hypothesis tests

0 of 12 correct

1.
For 60 heads in 100 fair tosses, $P(X \ge 60) = 0.02844$ . Find the two-sided p-value
2.
A test gives $p = 0.03$ . Is $H_0$ rejected at $\alpha = 0.05$ ? Type 1 for yes, 0 for no
3.
A test at $\alpha = 0.01$ is run on data for which $H_0$ is true. Find the probability of an error of the first kind
4.
A test has $\beta = 0.3$ for a given alternative. Find its power
5.
With $\sigma = 1$ known, 25 draws have $\bar{x} = 0.3$ . Find the z statistic for $H_0: \mu = 0$
6.
Sixteen values have $\bar{x} = 10.6$ and $s = 1.2$ . Find the t statistic for $H_0: \mu = 10$
7.
How many degrees of freedom does that t test use?
8.
The 95% t interval for $\mu$ is 2.1 to 3.9. Is $\mu_0 = 4$ rejected at the 5% level? Type 1 for yes, 0 for no
9.
A two-sided t test gives $t = 1.8$ and $p = 0.08$ . Find the one-sided p-value for $H_1: \mu > \mu_0$
10.
Twenty independent tests of true null hypotheses are run at $\alpha = 0.05$ . Find the chance that at least one rejects (an exact form or 4 figures)
11.
Ten hypotheses are tested with a Bonferroni correction for a family level of 0.05. At what level is each one tested?
12.
With $\sigma = 2$ , how many draws does the formula $\left(\frac{(1.96 + 0.84)\sigma}{\delta}\right)^2$ give for 80% power to detect $\delta = 0.5$ , before rounding up?

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

H_0

Hypothesis tests

Null and alternative hypotheses

The p-value

Rule: p-value

Rule: Significance level

Two kinds of error

Rule: Error rates and power

The t test for a mean

Rule: One-sample t test

Rule: Tests and intervals

One-sided tests

Rule: One-sided p-value

Many tests at once

Rule: Bonferroni correction

Common mistakes

Hypothesis tests

Summary card

Free further reading