5.7.3 The central limit theorem

The shape of a sum

Lesson 5.7.2 gave the centre and the spread of the sample mean $\bar{X}_n$ of $n$ i.i.d. draws: mean $\mu$ , standard error $\frac{\sigma}{\sqrt{n}}$ . The central limit theorem gives its shape. For large $n$ it is close to a normal, whatever the distribution of one draw.

To see the shape, remove the centre and the spread. Standardising (lesson 5.3.2) gives a variable with mean 0 and variance 1 for every $n$ . With $S_n = X_1 + \cdots + X_n$ the sum,

\begin{aligned} Z_n &= \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \\ &= \frac{S_n - n\mu}{\sigma\sqrt{n}} \end{aligned}

Rule: Central limit theorem

\begin{gathered} P(Z_n \le z) \to N(z) \text{ as } n \to \infty \\[4pt] \bar{X}_n \approx \mathcal{N}\!\left(\mu, \tfrac{\sigma^2}{n}\right) \\[4pt] S_n \approx \mathcal{N}(n\mu,\ n\sigma^2) \end{gathered}

For i.i.d. draws with mean $\mu$ and a finite variance $\sigma^2 > 0$ . $N$ is the standard normal distribution function. For large $n$ , the mean and the sum are approximately normal, with the mean and variance of lesson 5.7.2.

Without the standardising there is nothing to see: $\bar{X}_n$ collapses onto $\mu$ (lesson 5.7.2), and $S_n$ spreads out without limit. The theorem describes the shape at the scale $\frac{\sigma}{\sqrt{n}}$ .

For normal draws $Z_n$ is exactly standard normal at every $n$ (lesson 5.6.3). The theorem says that other distributions get there too, as $n$ grows.

Dice: from a triangle to a bell

Lesson 5.6.3 built the pmf of the sum of two dice by convolution: a triangle. Convolve in more dice and the triangle rounds into a bell.

Take 10 dice. The sum has mean $10 \times 3.5 = 35$ and standard deviation $\sqrt{10 \times 35/12} \approx 5.4006$ . By the exact pmf, $P(S_{10} \le 30) \approx 0.2050$ ; the normal gives $N\!\left(\frac{30 - 35}{5.4006}\right) \approx 0.1773$ .

Whole numbers: the half-step

The sum of dice takes whole values, and the normal spreads the probability of each value $s$ over the interval from $s - \frac{1}{2}$ to $s + \frac{1}{2}$ . So " $S \le 30$ " is best read as " $S \le 30.5$ ". This continuity correction gives $N\!\left(\frac{30.5 - 35}{5.4006}\right) \approx 0.2024$ , much closer to 0.2050.

With 100 dice the mean is 350 and the standard deviation about 17.0783. Exactly, $P(S_{100} \ge 380) \approx 0.0420$ . The normal gives 0.0395 from 380 and 0.0421 from the half-step at 379.5.

A coin works the same way. The number of heads in 100 tosses has mean 50 and standard deviation 5. Exactly, $P(\text{heads} \le 45) \approx 0.1841$ ; the normal gives 0.1587 from 45 and 0.1841 from 45.5.

Try it The loop convolves in one die at a time, 100 times (lesson 5.6.3), and stops to check 10 dice on the way. Each line prints the standard deviation, the exact probability, the normal, and the normal with the half-step. The last line is the coin, with binom.cdf for the exact answer.

import math import numpy as np from scipy.stats import binom, norm die = np.array([0] + [1 / 6] * 6) pmf = np.array([1.0]) for k in range(1, 101): pmf = np.convolve(pmf, die) if k == 10: sd = math.sqrt(10 * 35 / 12) print(10, round(sd, 4), round(pmf[:31].sum(), 4), round(norm.cdf((30 - 35) / sd), 4), round(norm.cdf((30.5 - 35) / sd), 4)) sd = math.sqrt(100 * 35 / 12) print(100, round(sd, 4), round(pmf[380:].sum(), 4), round(norm.sf((380 - 350) / sd), 4), round(norm.sf((379.5 - 350) / sd), 4)) print(round(binom.cdf(45, 100, 0.5), 4), round(norm.cdf((45 - 50) / 5), 4), round(norm.cdf((45.5 - 50) / 5), 4))

⌘/Ctrl + Enter runs the cell

Three starting distributions

The theorem does not care where the draws come from. Three very different starts:

Uniform on $[0, 1]$ : flat and symmetric, with $\mu = \frac{1}{2}$ and $\sigma^2 = \frac{1}{12}$ .
Exponential with mean 1 (lesson 5.4.2): skewed, with a long right tail, $\mu = 1$ and $\sigma = 1$ .
A fair coin, 0 or 1: only two values, $\mu = \frac{1}{2}$ and $\sigma = \frac{1}{2}$ .

The standardised mean of 10 draws from each start. The uniform's is already almost normal, and the coin's follows the curve in steps. The exponential's is still skewed, so its upper tail holds more than the normal's 0.025.

A simulation shows all three at once. For each start and each $n$ , draw 20,000 samples of size $n$ and standardise their means.

Then count the share within $\pm 1.96$ , 95% for a normal, and the share above 1.96, 2.5% for a normal. In one seeded run:

start	$n = 1$	$n = 5$	$n = 30$
uniform, share within $\pm 1.96$	1	0.9514	0.9508
exponential, within $\pm 1.96$	0.9466	0.957	0.9526
exponential, above 1.96	0.0534	0.0427	0.0338
coin, within $\pm 1.96$	1	0.9386	0.9556

A single uniform draw never strays 1.96 standard deviations, and a single coin is always exactly one away. By $n = 5$ the uniform is already close to 95%. The exponential's total is near 95% even at $n = 1$ , but lopsided: too much above 1.96, too little below $-1.96$ .

n

import math import numpy as np from scipy.stats import skew rng = np.random.default_rng(5734) bases = { "uniform": (lambda size: rng.random(size), 0.5, math.sqrt(1 / 12)), "exponential": (lambda size: rng.exponential(1, size), 1, 1), "coin": (lambda size: rng.integers(0, 2, size), 0.5, 0.5), } for name, (draw, mu, sigma) in bases.items(): for n in [1, 5, 30]: means = draw((20_000, n)).mean(axis=1) z = (means - mu) / (sigma / math.sqrt(n)) inside = np.mean(np.abs(z) <= 1.96) print(name, n, round(float(inside), 4), round(float(np.mean(z > 1.96)), 4), round(float(skew(means)), 2) + 0.0)

⌘/Ctrl + Enter runs the cell

How fast: the skewness fades like $\frac{1}{\sqrt{n}}$

The skewness of a mean of $n$ draws is the skewness of one draw divided by $\sqrt{n}$ (stated here). The uniform and the coin have skewness 0, so only their flatness or gaps must fade.

The exponential starts at 2, and its mean is still visibly skewed long after. The simulation above found 2.01, 0.87 and 0.34 at $n$ = 1, 5 and 30, against $\frac{2}{\sqrt{n}}$ = 2, 0.8944 and 0.3651.

The exponential's sum is a gamma distribution (lesson 5.6.3 met the case $n = 2$ ), so its tails are known exactly:

$n$	1	5	10	30	100
$P(Z_n > 1.96)$	0.0518	0.0433	0.0393	0.034	0.0302
$P(Z_n < -1.96)$	0	0.0004	0.0058	0.014	0.0192
skewness $\frac{2}{\sqrt{n}}$	2	0.8944	0.6325	0.3651	0.2

Even at $n = 100$ the upper tail holds 3.02%, not 2.5%. The more skewed the draws, the larger $n$ must be.

n

import math from scipy.stats import gamma for n in [1, 5, 10, 30, 100]: hi, lo = n + 1.96 * math.sqrt(n), n - 1.96 * math.sqrt(n) print(n, round(gamma.sf(hi, n), 4), round(gamma.cdf(lo, n), 4), round(2 / math.sqrt(n), 4))

⌘/Ctrl + Enter runs the cell

Set the explorer to $n = 1$ : the histogram is the exponential itself. Step through $n = 2$ , 5, 30 and 100. The histogram narrows like $\frac{1}{\sqrt{n}}$ and its skew fades.

The share within $1.96\frac{\sigma}{\sqrt{n}}$ in the readout stays near 95% throughout; the histogram shows how lopsided it is for small $n$ .

Explore

Each bar counts how many of 2,000 samples had their mean in that range. Raise the sample size n: the histogram narrows like 1/√n and takes the shape of the normal curve, even from a skewed start.

Sample size n = 5

True mean and standard deviation: μ = 1, σ = 1
The 2000 sample means: average 1.0092, standard deviation 0.4566 against σ/√n = 0.4472
Within 1.96 σ/√n of μ: 95.3% of the means (the normal curve says 95%)

The $\frac{1}{\sqrt{n}}$ scale of the error

By the theorem, about 95% of sample means land within $1.96\frac{\sigma}{\sqrt{n}}$ of $\mu$ : the 1.96 is the normal's quantile from lesson 5.4.3. That band has half-width proportional to $\frac{1}{\sqrt{n}}$ .

Rule: Error scale of a mean

\begin{gathered} P\left(|\bar{X}_n - \mu| \le 1.96\,\frac{\sigma}{\sqrt{n}}\right) \approx 0.95 \\[4pt] n \approx \left(\frac{1.96\,\sigma}{\varepsilon}\right)^2 \end{gathered}

The second line is the number of draws for a 95% half-width $\varepsilon$ . Ten times the draws narrow the band $\sqrt{10} \approx 3.16$ times. One more correct digit, a band ten times narrower, needs a hundred times the draws.

For a die the 95% half-width is 0.3347 after 100 rolls, 0.1059 after 1,000 and 0.0335 after 10,000. To get within 0.1 with 95% probability takes $(1.96\sigma/0.1)^2$ , about 1,121 rolls. Chebyshev's inequality, which must cover every distribution, asked for 5,834 (lesson 5.7.2).

Case study 3.7.4's band

Case study 3.7.4 estimated a correlation 400 times at each sample size. The middle 90% of the estimates ran from 0.363 to 0.604 with 100 draws, 0.462 to 0.537 with 1,000, and 0.487 to 0.513 with 10,000.

The widths are 0.241, 0.075 and 0.026. Each tenfold step narrowed the band 3.21 and then 2.88 times: about $\sqrt{10} \approx 3.16$ , as the rule says.

An estimated correlation is not a plain mean, but it is built from means of products and squares of the draws. For large samples its error behaves like a mean's error: close to normal, with a spread proportional to $\frac{1}{\sqrt{n}}$ .

Case study 4.7.5's standard errors

Case study 4.7.5 found each simulated probability within two standard errors $\sqrt{\hat{p}(1 - \hat{p})/N}$ of the truth. The theorem says how often that happens.

The share $\hat{p}$ is a mean of indicators (lesson 5.7.2), so it is close to normal. A miss within 1.96 standard errors then has probability about 0.95.

Repeat case study 4.7.5's simulation with $N = 10{,}000$ draws 100,000 times, with a seed. The miss was within 1.96 estimated standard errors 94.67% of the time, and within 2 of them 95.39%. A simulated number with its standard error is a statement of this kind.

\sqrt{10}

import math import numpy as np sigma = math.sqrt(35 / 12) for n in [100, 1000, 10_000]: print(n, round(1.96 * sigma / math.sqrt(n), 4)) print(round(math.sqrt(10), 4), math.ceil((1.96 * sigma / 0.1) ** 2)) widths = [0.604 - 0.363, 0.537 - 0.462, 0.513 - 0.487] print([round(w, 3) for w in widths], round(widths[0] / widths[1], 2), round(widths[1] / widths[2], 2)) p, N = 0.028810, 10_000 rng = np.random.default_rng(5732) phat = rng.binomial(N, p, size=100_000) / N se = np.sqrt(phat * (1 - phat) / N) print(round(float(np.mean(np.abs(phat - p) <= 1.96 * se)), 4), round(float(np.mean(np.abs(phat - p) <= 2 * se)), 4))

⌘/Ctrl + Enter runs the cell

Where the theorem fails

Far in the tails. The theorem is about the middle of the distribution. Even where the middle is close, a tail probability can be off many times over.

For the exponential at $n = 30$ , $P(Z_n > 4) \approx 0.000386$ , against the normal's 0.000032: 12.2 times as large. At $n = 100$ it is still 5.1 times.

No finite variance. A Student t with $\nu = 1$ has neither a variance nor a mean (lesson 5.4.4). The mean of $n$ such draws has the same distribution as one draw (stated here), so averaging narrows nothing.

In a seeded simulation the middle 90% of such means had widths 13.6, 13.86, 14.42 and 11.79 for $n$ = 1, 10, 100 and 1,000. One draw's exact width is 12.63.

Dependence. The theorem needs independent draws, or nearly so. Copies of one draw, or strongly correlated draws, do not average out at the rate $\frac{\sigma}{\sqrt{n}}$ .

P(Z_n > 4)

import math import numpy as np from scipy.stats import gamma, norm, t for n in [30, 100]: print(n, round(gamma.sf(n + 4 * math.sqrt(n), n), 6), round(norm.sf(4), 6), round(gamma.sf(n + 4 * math.sqrt(n), n) / norm.sf(4), 1)) rng = np.random.default_rng(5733) for n in [1, 10, 100, 1000]: means = rng.standard_t(1, size=(2_000, n)).mean(axis=1) lo, hi = np.percentile(means, [5, 95]) print(n, round(float(hi - lo), 2)) print(round(2 * t.ppf(0.95, 1), 2))

⌘/Ctrl + Enter runs the cell

Case study 5.7.5 uses the theorem to put a 95% interval around a simulated option price.

Common mistakes

The theorem is about sums and means, not about the draws. A thousand exponential draws are still skewed; their mean is close to normal.
Standardise with the standard error $\frac{\sigma}{\sqrt{n}}$ , not with $\sigma$ of one draw.
$n = 30$ is not a magic number. For the exponential at $n = 30$ the upper tail beyond 1.96 holds 0.034, not 0.025.
Whole-number counts need the half-step. For heads in 100 tosses, $P(\text{heads} \le 45)$ uses 45.5: 0.1841, not 0.1587.
Ten times the draws narrow a band $\sqrt{10} \approx 3.16$ times, not 10.
No finite variance, no theorem. Means of t draws with $\nu = 1$ never narrow.
Do not trust the normal far in the tails. The middle can be close while a 4-sigma probability is off by a factor of 12.

The central limit theorem

0 of 11 correct

1.
Find the mean of the sum of 100 fair dice
2.
Find the variance of the sum of 100 fair dice, as a fraction
3.
The sum of 48 independent uniforms on $[0, 1]$ has mean 24. Given $N(1) \approx 0.8413$ , estimate $P(S > 26)$
4.
Draws have $\mu = 10$ and $\sigma = 5$ . Given $N(2) \approx 0.9772$ , estimate $P(\bar{X}_{25} > 12)$
5.
Draws have $\sigma = 2$ . Find the 95% half-width $1.96\frac{\sigma}{\sqrt{n}}$ for the mean of 100 of them
6.
Draws have $\sigma = 2$ . How many make the 95% half-width $1.96\frac{\sigma}{\sqrt{n}}$ equal to 0.196?
7.
By what factor does a 95% band narrow when the number of draws is multiplied by 10? (type sqrt)
8.
A band has width 0.241 with 100 draws. Predict its width with 10,000 draws
9.
A fair coin is tossed 100 times. Given $N(0.9) \approx 0.8159$ , estimate $P(\text{heads} \le 45)$ with the half-step
10.
One exponential draw has skewness 2. Find the skewness of the mean of 25 such draws
11.
A fair coin is tossed 400 times. Given $N(2) \approx 0.9772$ , estimate the probability that the share of heads exceeds 0.55

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}}

The central limit theorem

The shape of a sum

Rule: Central limit theorem

Dice: from a triangle to a bell

Whole numbers: the half-step

Three starting distributions

How fast: the skewness fades like $\frac{1}{\sqrt{n}}$

The $\frac{1}{\sqrt{n}}$ scale of the error

Rule: Error scale of a mean

Case study 3.7.4's band

Case study 4.7.5's standard errors

Where the theorem fails

Common mistakes

The central limit theorem

Summary card

Free further reading

The shape of a sum

Rule: Central limit theorem

Dice: from a triangle to a bell

Whole numbers: the half-step

Three starting distributions

How fast: the skewness fades like 1n\frac{1}{\sqrt{n}}n​1​

The 1n\frac{1}{\sqrt{n}}n​1​ scale of the error

Rule: Error scale of a mean

Case study 3.7.4's band

Case study 4.7.5's standard errors

Where the theorem fails

Common mistakes

Summary card

Free further reading

How fast: the skewness fades like $\frac{1}{\sqrt{n}}$

The $\frac{1}{\sqrt{n}}$ scale of the error