5.8.2 Sampling distributions and confidence intervals

The sampling distribution of the mean

Lesson 5.8.1 found that the sample mean $\bar{X}$ of $n$ independent draws has mean $\mu$ and variance $\frac{\sigma^2}{n}$ . Its distribution over all possible samples is its sampling distribution. Its standard deviation has its own name.

Rule: Standard error of the mean

\begin{gathered} \operatorname{SE}(\bar{X}) = \frac{\sigma}{\sqrt{n}} \\[4pt] \widehat{\operatorname{SE}} = \frac{s}{\sqrt{n}} \end{gathered}

The standard error of an estimator is its standard deviation: the typical size of its error. It is in the units of the data. When $\sigma$ is unknown, the sample standard deviation $s$ of lesson 5.8.1 stands in for it.

The shape is known too. For normal draws, $\bar{X} \sim \mathcal{N}(\mu, \frac{\sigma^2}{n})$ exactly (lesson 5.6.3). For any other distribution with a finite variance it is close to that for large $n$ , by the central limit theorem.

An example. Exponential draws with mean 1 have $\sigma = 1$ (lesson 5.4.2), so the mean of 30 has standard error $\frac{1}{\sqrt{30}} \approx 0.1826$ . The total of 30 exponentials has a gamma distribution (lesson 5.6.3 met the total of two), so the exact answers are known:

	exact	normal approximation	simulated
$P(\bar{X} \le 1)$	0.5243	0.5	0.5253
$P(\lvert \bar{X} - 1 \rvert \le 1.96\,\operatorname{SE})$	0.9519	0.95	0.9509

The skew of the exponential leaves a trace: slightly more than half the means fall below $\mu$ . But the band of $\pm 1.96$ standard errors holds almost exactly 95% of them.

Two case studies used this rule already. In case study 5.3.5, 1,000 daily returns with volatility 1% have a mean with standard error $\frac{1\%}{\sqrt{1000}} \approx 0.0316\%$ . Case study 4.6.5 quoted $\frac{\sigma_i}{\sqrt{T}}$ for an annual mean from $T$ years. Each year is one draw with standard deviation $\sigma_i$ , so $n = T$ ; for B that is $\frac{0.18}{\sqrt{10}} \approx 0.0569$ .

Try it gamma(30, scale=1/30) is the exact distribution of the mean of 30 exponentials with mean 1. The first line prints the standard error and the two exact probabilities. Then 100,000 seeded samples of 30: the mean and standard deviation of their means, and the two shares. The last line is the two case studies' standard errors.

import numpy as np from scipy.stats import gamma n = 30 se = 1 / np.sqrt(n) exact = gamma(n, scale=1 / n) print(round(se, 4), round(exact.cdf(1), 4), round(exact.cdf(1 + 1.96 * se) - exact.cdf(1 - 1.96 * se), 4)) means = np.random.default_rng(582).exponential(1.0, size=(100_000, n)).mean(axis=1) print(round(means.mean(), 4), round(means.std(), 4), round(np.mean(means <= 1), 4), round(np.mean(np.abs(means - 1) <= 1.96 * se), 4)) print(round(100 * 0.01 / np.sqrt(1000), 4), round(0.18 / np.sqrt(10), 4))

⌘/Ctrl + Enter runs the cell

A confidence interval when $\sigma$ is known

Standardise the mean: $Z = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}}$ is standard normal, exactly or approximately. So $P(-1.96 \le Z \le 1.96) = 0.95$ (lesson 5.4.3). Rearranged, the same event says that $\mu$ lies within $1.96\,\frac{\sigma}{\sqrt{n}}$ of $\bar{X}$ .

Rule: Confidence interval for μ, σ known

\begin{gathered} \bar{x} \pm z\,\frac{\sigma}{\sqrt{n}} \\[4pt] z = 1.645,\ 1.96,\ 2.576 \\ \text{for } 90\%,\ 95\%,\ 99\% \end{gathered}

For a confidence level $1 - \alpha$ , $z$ is norm.ppf(1 - α/2), leaving $\frac{\alpha}{2}$ in each tail. The interval runs from $\bar{x} - z\frac{\sigma}{\sqrt{n}}$ to $\bar{x} + z\frac{\sigma}{\sqrt{n}}$ ; half its width is the margin of error.

Roll a die 30 times; its $\sigma = \sqrt{35/12} \approx 1.7078$ is known. One seeded sample has $\bar{x} \approx 3.7333$ . The margin at 95% is $1.96 \times \frac{1.7078}{\sqrt{30}} \approx 0.6111$ , so the interval 3.1222 to 4.3445 contains the true mean 3.5.

More confidence costs width. At 90% the margin is 0.5129; at 99% it is 0.8032.
More data buys width slowly. The margin shrinks like $\frac{1}{\sqrt{n}}$ : four times the rolls, 120, halve it to 0.3056.

z

import numpy as np from scipy.stats import norm sigma, n = np.sqrt(35 / 12), 30 x = np.random.default_rng(5830).integers(1, 7, size=n) for level in [0.90, 0.95, 0.99]: z = norm.ppf(1 - (1 - level) / 2) margin = z * sigma / np.sqrt(n) print(level, round(z, 3), round(margin, 4), round(x.mean() - margin, 4), round(x.mean() + margin, 4)) print(round(sigma, 4), round(x.mean(), 4), round(1.96 * sigma / np.sqrt(120), 4))

⌘/Ctrl + Enter runs the cell

What "95%" means

Before the rolls, the interval is random: its centre $\bar{X}$ changes from sample to sample. The 95% is the chance that this procedure produces an interval that covers $\mu$ . It is called the interval's coverage.

After the rolls, the interval from 3.1222 to 4.3445 is fixed, and so is $\mu$ . Either it contains $\mu$ or it does not; nothing random is left. "95% confidence" describes the method, not this one interval.

In one seeded run of 10,000 samples of 30 rolls, 95.58% of the intervals cover 3.5. The exact share, from the distribution of the total of 30 dice, is 0.9525. It is a little above 95% because that total moves in whole steps.

The explorer starts with the die. Set $n = 30$ : the share of means within $1.96\,\sigma/\sqrt{n}$ of $\mu$ is the share of intervals $\bar{x} \pm 1.96\,\sigma/\sqrt{n}$ that cover $\mu$ . Then try the coin with small $n$ : its mean takes only a few values, and the share jumps around 95%.

Explore

Each bar counts how many of 2,000 samples had their mean in that range. Raise the sample size n: the histogram narrows like 1/√n and takes the shape of the normal curve, even from a skewed start.

Sample size n = 5

True mean and standard deviation: μ = 3.5, σ = 1.7078
The 2000 sample means: average 3.5223, standard deviation 0.7615 against σ/√n = 0.7638
Within 1.96 σ/√n of μ: 94% of the means (the normal curve says 95%)

Try it The first lines draw 10,000 seeded samples of 30 rolls and print the share of intervals that cover 3.5. The loop builds the exact distribution of the total of 30 dice by convolution, one die at a time (lesson 5.6.3). The last line adds up the totals whose interval covers 3.5.

import numpy as np sigma, n = np.sqrt(35 / 12), 30 margin = 1.96 * sigma / np.sqrt(n) means = np.random.default_rng(5831).integers(1, 7, size=(10_000, n)).mean(axis=1) print(np.mean(np.abs(means - 3.5) <= margin)) total = np.array([1.0]) for _ in range(n): total = np.convolve(total, np.ones(6) / 6) S = np.arange(n, 6 * n + 1) print(round(total[np.abs(S / n - 3.5) <= margin].sum(), 4))

⌘/Ctrl + Enter runs the cell

When $\sigma$ is unknown: the t interval

In practice $\sigma$ is unknown, and the sample standard deviation $s$ takes its place. But $s$ varies from sample to sample, and now and then it is much too small. Then the standardised mean is too large, so it has fatter tails than a normal.

Rule: The standardised mean with s

T = \frac{\bar{X} - \mu}{s/\sqrt{n}} \sim t_{n - 1}

For normal draws, $T$ has the Student t distribution with $n - 1$ degrees of freedom (lesson 5.4.4), the same $n - 1$ that $s$ divides by. For other distributions it is approximately $t_{n-1}$ for large $n$ .

This is lesson 5.4.4's ratio $\frac{Z}{\sqrt{V/\nu}}$ , with $\nu = n - 1$ . Divide top and bottom by $\sigma/\sqrt{n}$ :

\begin{gathered} T = \frac{Z}{\sqrt{V/(n - 1)}} \\[4pt] Z = \frac{\bar{X} - \mu}{\sigma/\sqrt{n}} \\[4pt] V = \frac{(n - 1)s^2}{\sigma^2} \end{gathered}

For normal draws $V$ is $\chi^2_{n-1}$ and independent of $Z$ (stated here); the next section uses $V$ .

Rule: t interval for μ

\begin{gathered} \bar{x} \pm t_{n-1}\,\frac{s}{\sqrt{n}} \\[4pt] t_{n-1} = \texttt{t.ppf(0.975, n - 1)} \end{gathered}

The 95% interval when $\sigma$ is estimated. For 90% or 99%, use the 0.95 or 0.995 quantile.

$n$	2	5	10	30	100	normal
$t_{n-1}$ , 0.975-quantile	12.706	2.776	2.262	2.045	1.984	1.960

For $n = 5$ the multiplier is 2.776 instead of 1.96, as lesson 5.4.4 found: small samples need wider intervals. From about 30 draws on, the difference is small.

Five measurements. A quantity is measured five times: 10.2, 9.6, 10.9, 10.4 and 9.9. Then $\bar{x} = 10.2$ , $s^2 = 0.245$ , $s \approx 0.495$ , and the standard error is $\frac{0.495}{\sqrt{5}} \approx 0.2214$ . The margin is $2.776 \times 0.2214 \approx 0.6146$ , so the 95% interval runs from 9.5854 to 10.8146.

With 1.96 in place of 2.776, the margin would be 0.4339: too narrow. The coverage of such an interval is $P(\lvert T \rvert \le 1.96)$ for $T \sim t_{n-1}$ :

$n$	5	10	30
coverage of $\bar{x} \pm 1.96\,\frac{s}{\sqrt{n}}$	0.8784	0.9184	0.9403
coverage of the t interval	0.95	0.95	0.95

The same 100 samples of five, with two multipliers. Highlighted intervals miss the true value 10: 5 with the t quantile, 10 with 1.96. Each interval's length depends on its own $s$ .

Try it The first lines compute the five-measurement interval by hand. t.interval is SciPy's shortcut for the same interval. Then the t quantiles of the table.

import numpy as np from scipy.stats import t x = np.array([10.2, 9.6, 10.9, 10.4, 9.9]) n = len(x) xbar, s = x.mean(), x.std(ddof=1) se, q = s / np.sqrt(n), t.ppf(0.975, n - 1) print(round(xbar, 4), round(s**2, 4), round(s, 4), round(se, 4), round(q, 3)) print(round(xbar - q * se, 4), round(xbar + q * se, 4), round(1.96 * se, 4)) print(np.round(t.interval(0.95, n - 1, loc=xbar, scale=se), 4)) print([round(float(t.ppf(0.975, m - 1)), 3) for m in [2, 5, 10, 30, 100]])

⌘/Ctrl + Enter runs the cell

Data that are not normal

The t interval assumes normal draws. For other data it relies on the central limit theorem, so it works for large samples and can fall short for small, skewed ones. For exponential draws its coverage is 90.09%, 92.72% and 94.18% at $n$ = 10, 30 and 100, in seeded runs.

\pm 1.96\,s/\sqrt{n}

import numpy as np from scipy.stats import t for n in [5, 10, 30]: print(n, round(1 - 2 * t.sf(1.96, n - 1), 4)) X = np.random.default_rng(5822).normal(10, 0.5, size=(100_000, 5)) T = (X.mean(axis=1) - 10) / (X.std(axis=1, ddof=1) / np.sqrt(5)) print(round(np.mean(np.abs(T) <= 1.96), 4), round(np.mean(np.abs(T) <= t.ppf(0.975, 4)), 4)) rng = np.random.default_rng(5823) for n in [10, 30, 100]: E = rng.exponential(1.0, size=(50_000, n)) T = (E.mean(axis=1) - 1) / (E.std(axis=1, ddof=1) / np.sqrt(n)) print(n, round(np.mean(np.abs(T) <= t.ppf(0.975, n - 1)), 4))

⌘/Ctrl + Enter runs the cell

An interval for a variance

The $V$ above gives an interval for $\sigma^2$ itself. For normal draws, $V = \frac{(n - 1)s^2}{\sigma^2}$ lies between the 0.025- and 0.975-quantiles of $\chi^2_{n-1}$ with probability 0.95. Solving both ends for $\sigma^2$ gives the interval.

Rule: Interval for σ², normal data

\begin{gathered} \frac{(n - 1)s^2}{\chi^2_{0.975}} \le \sigma^2 \le \frac{(n - 1)s^2}{\chi^2_{0.025}} \end{gathered}

$\chi^2_p$ is chi2.ppf(p, n - 1). Dividing by the larger quantile gives the lower end. Take square roots of both ends for an interval for $\sigma$ .

For the five measurements, $(n - 1)s^2 = 0.98$ and the quantiles of $\chi^2_4$ are 0.4844 and 11.1433. So $\sigma^2$ lies between 0.0879 and 2.023, and $\sigma$ between 0.2966 and 1.4223.

The estimate $s^2 = 0.245$ is far from the middle: the interval stretches to the right, because $\chi^2$ is skewed (lesson 5.4.4). Five values say little about a spread.

This interval needs normal data much more than the t interval does. A fat tail (lesson 5.3.3) makes $s^2$ swing more than $\chi^2$ allows.

Take samples of 30, in one seeded run. The coverage is 94.99% for normal draws, but 82.54% for a Student t with 5 degrees of freedom.

\sigma^2

import numpy as np from scipy.stats import chi2 x = np.array([10.2, 9.6, 10.9, 10.4, 9.9]) n, s2 = len(x), x.var(ddof=1) lo, hi = chi2.ppf([0.025, 0.975], n - 1) print(round((n - 1) * s2, 4), round(lo, 4), round(hi, 4)) print(round((n - 1) * s2 / hi, 4), round((n - 1) * s2 / lo, 4)) print(round(np.sqrt((n - 1) * s2 / hi), 4), round(np.sqrt((n - 1) * s2 / lo), 4)) rng = np.random.default_rng(5824) a, b = chi2.ppf([0.025, 0.975], 29) for data, true in [(rng.standard_normal((50_000, 30)), 1.0), (rng.standard_t(5, size=(50_000, 30)), 5 / 3)]: v = data.var(axis=1, ddof=1) print(round(np.mean((29 * v / b <= true) & (true <= 29 * v / a)), 4))

⌘/Ctrl + Enter runs the cell

The bootstrap

The intervals so far needed a formula for the sampling distribution. When there is none, the bootstrap simulates it from the data. It treats the sample as if it were the whole population, and draws new samples from it.

Rule: Bootstrap percentile interval

\hat{\theta}^*_1, \ldots, \hat{\theta}^*_B

Draw $n$ values from the sample with replacement, so some repeat and some are left out, and compute the estimate $\hat{\theta}^*$ . Repeat $B$ times, say 10,000. The 0.025- and 0.975-quantiles of the $B$ estimates form a 95% interval.

Take 40 seeded exponential draws with mean 1. Their mean is 1.0984 and $s = 1.3211$ . The median, the middle value, is 0.7425; the true median of this distribution is $\ln 2 \approx 0.6931$ .

statistic	estimate	bootstrap 95% interval	formula interval
mean	1.0984	0.7287 to 1.5262	t: 0.6759 to 1.5209
median	0.7425	0.2667 to 1.0386	none in this module

For the mean, the standard deviation of the 10,000 bootstrap means is 0.2055, close to $\frac{s}{\sqrt{n}} \approx 0.2089$ . The bootstrap interval is not centred on 1.0984: it leans right, as the skewed data do. For the median, the bootstrap gives an interval where this module has no formula.

The bootstrap has limits.

It needs independent draws. For a time series, whose values depend on their neighbours, resample blocks of neighbouring values instead.
It cannot see what the sample lacks. A small sample without its distribution's rare extremes gives bootstrap samples without them too.
It is a simulation. Use many resamples and a fixed seed, so the interval does not move from run to run.

Case study 5.8.5 bootstraps a Sharpe ratio.

Try it idx holds 10,000 rows of 40 positions drawn with replacement, and data[idx] the resampled values. Each row's mean and median is one bootstrap estimate. The lines print the sample's statistics, the bootstrap interval and standard deviation of the mean, the t interval, and the bootstrap interval of the median.

import numpy as np from scipy.stats import t data = np.random.default_rng(5829).exponential(1.0, size=40) n = len(data) print(round(data.mean(), 4), round(data.std(ddof=1), 4), round(np.median(data), 4)) idx = np.random.default_rng(58290).integers(0, n, size=(10_000, n)) boot_mean, boot_median = data[idx].mean(axis=1), np.median(data[idx], axis=1) print(np.round(np.quantile(boot_mean, [0.025, 0.975]), 4), round(boot_mean.std(), 4), round(data.std(ddof=1) / np.sqrt(n), 4)) margin = t.ppf(0.975, n - 1) * data.std(ddof=1) / np.sqrt(n) print(round(data.mean() - margin, 4), round(data.mean() + margin, 4)) print(np.round(np.quantile(boot_median, [0.025, 0.975]), 4), round(np.log(2), 4))

⌘/Ctrl + Enter runs the cell

Lesson 5.8.3 turns the same standardised mean into a test of a claimed value of $\mu$ .

Common mistakes

"There is a 95% chance that $\mu$ lies between 9.5854 and 10.8146." Not so: once computed, the interval contains $\mu$ or does not. The 95% belongs to the method.
Using 1.96 with $s$ in a small sample. With five values it covers only 87.84% of the time; use $t_{n-1}$ .
Confusing standard deviation and standard error. $s$ measures the spread of single draws; $\frac{s}{\sqrt{n}}$ measures the spread of their mean.
Expecting twice the data to halve the interval. It takes four times the data.
Reading the variance interval as symmetric. It stretches to the right, and it is reliable only for normal data.
Expecting the bootstrap to add information. It reuses the sample, so it cannot fix a small or unrepresentative one.

Sampling distributions and confidence intervals

0 of 12 correct

1.
Draws have standard deviation $\sigma = 2$ . Find the standard error of the mean of 16 draws
2.
A sample of 36 has $s = 3$ . Find the estimated standard error of its mean
3.
With $\sigma = 10$ known, 25 draws have $\bar{x} = 50$ . Find the upper end of the 95% interval for $\mu$
4.
An interval from 25 draws has margin 4. With $\sigma$ known, how many draws give margin 2?
5.
A 95% interval with $\sigma$ known has margin $1.96 \times 2 = 3.92$ . Find the margin of the 99% interval, with $z = 2.576$
6.
A procedure gives 95% intervals. Over 200 independent samples, how many intervals do you expect to miss $\mu$ ?
7.
How many degrees of freedom does the t interval from 12 values use?
8.
Five values have $\bar{x} = 20$ and $s = 1$ ; $t_4 = 2.776$ . Find the margin of the 95% t interval (type sqrt)
9.
With the same $s$ , by what factor is a t interval from 5 values wider than one using 1.96? ( $t_4 = 2.776$ )
10.
Normal samples of 10 have $V = \frac{9s^2}{\sigma^2} \sim \chi^2_9$ . Find $\operatorname{Var}(V)$
11.
Five normal values have $s^2 = 2$ . With $\chi^2_{0.025} = 0.4844$ for 4 degrees of freedom, find the upper end of the 95% interval for $\sigma^2$
12.
10,000 bootstrap means are sorted from smallest to largest. Which one, counting from the smallest, is the 2.5% quantile? Give the position

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\frac{\sigma}{\sqrt{n}}

Sampling distributions and confidence intervals

The sampling distribution of the mean

Rule: Standard error of the mean

A confidence interval when $\sigma$ is known

Rule: Confidence interval for μ, σ known

What "95%" means

When $\sigma$ is unknown: the t interval

Rule: The standardised mean with s

Rule: t interval for μ

Data that are not normal

An interval for a variance

Rule: Interval for σ², normal data

The bootstrap

Rule: Bootstrap percentile interval

Common mistakes

Sampling distributions and confidence intervals

Summary card

Free further reading

The sampling distribution of the mean

Rule: Standard error of the mean

A confidence interval when σ\sigmaσ is known

Rule: Confidence interval for μ, σ known

What "95%" means

When σ\sigmaσ is unknown: the t interval

Rule: The standardised mean with s

Rule: t interval for μ

Data that are not normal

An interval for a variance

Rule: Interval for σ², normal data

The bootstrap

Rule: Bootstrap percentile interval

Common mistakes

Summary card

Free further reading

A confidence interval when $\sigma$ is known

When $\sigma$ is unknown: the t interval