5.7.2 The law of large numbers

The sample mean is a random variable

Roll a die 10 times and average the results. Roll 10 more and the average changes. Before the rolls are made, their average is a random variable, with a distribution of its own.

Write $X_1, \ldots, X_n$ for $n$ draws that are independent and identically distributed, or i.i.d.: independent (lesson 5.1.3), and each with the distribution of one draw $X$ . That draw has mean $\mu$ and variance $\sigma^2$ .

The sample mean is

\bar{X}_n = \frac{X_1 + X_2 + \cdots + X_n}{n}

Rule: Mean and variance of the sample mean

\begin{gathered} \mathbb{E}[\bar{X}_n] = \mu \\[4pt] \operatorname{Var}(\bar{X}_n) = \frac{\sigma^2}{n} \\[4pt] \sigma_{\bar{X}_n} = \frac{\sigma}{\sqrt{n}} \end{gathered}

By linearity the sum has mean $n\mu$ ; dividing by $n$ gives $\mu$ . The variances of independent draws add up to $n\sigma^2$ , and dividing by $n$ divides the variance by $n^2$ (lesson 5.3.2). The standard deviation $\frac{\sigma}{\sqrt{n}}$ is called the standard error of the mean: the typical size of the miss $\bar{X}_n - \mu$ .

For a die, $\mu = 3.5$ and $\sigma^2 = \frac{35}{12} \approx 2.9167$ :

rolls $n$	1	10	100	1,000	10,000
$\operatorname{Var}(\bar{X}_n)$	2.916667	0.291667	0.029167	0.002917	0.000292
$\frac{\sigma}{\sqrt{n}}$	1.7078	0.5401	0.1708	0.054	0.0171

Four times the rolls halve the standard error, and a hundred times divide it by 10. The variance falls like $\frac{1}{n}$ ; the standard error only like $\frac{1}{\sqrt{n}}$ .

Lesson 5.6.3 met this $\frac{\sigma^2}{n}$ for normals, where the mean of $n$ draws is exactly $\mathcal{N}\left(\mu, \frac{\sigma^2}{n}\right)$ . The mean and variance hold for any distribution; the shape is lesson 5.7.3's subject. Case study 5.3.5 used the same rule: with $\sigma = 1\%$ , the mean of 1,000 daily returns has standard error $\frac{1\%}{\sqrt{1000}} \approx 0.0316\%$ .

n

import math import numpy as np var = 35 / 12 for n in [1, 10, 100, 1000, 10_000]: print(n, round(var / n, 6), round(math.sqrt(var / n), 4)) rng = np.random.default_rng(5721) for n in [10, 100, 1000]: means = rng.integers(1, 7, size=(2_000, n)).mean(axis=1) print(n, round(float(means.mean()), 4), round(float(means.std()), 4)) print(round(1 / math.sqrt(1000), 4))

⌘/Ctrl + Enter runs the cell

The law of large numbers

With a spread that shrinks to 0, the sample mean must close in on $\mu$ . Chebyshev's inequality (lesson 5.7.1), applied to $\bar{X}_n$ with its variance $\frac{\sigma^2}{n}$ , makes that precise.

Rule: Law of large numbers

\begin{gathered} P\big(|\bar{X}_n - \mu| \ge \varepsilon\big) \le \frac{\sigma^2}{n\,\varepsilon^2} \\[4pt] \to 0 \text{ as } n \to \infty \end{gathered}

For i.i.d. draws with mean $\mu$ and finite variance $\sigma^2$ , and any distance $\varepsilon > 0$ . However small a miss you name, the chance of missing by that much shrinks to 0. The law still holds when the variance is infinite, as long as the mean exists; then the bound is not available.

The bound also says how many draws are enough. For a die and a miss of $\varepsilon = 0.1$ , it is $\frac{2.9167}{0.01\,n}$ : 0.2917 with 1,000 rolls and 0.0292 with 10,000.

The true chance is smaller. In 4,000 seeded repetitions of 1,000 rolls, the share that missed 3.5 by 0.1 or more was 0.0712, against the bound's 0.2917. Chebyshev's inequality must hold for every distribution, so it is cautious.

To push the bound down to $\delta = 0.05$ , solve $\frac{\sigma^2}{n\varepsilon^2} \le \delta$ : $n \ge \frac{\sigma^2}{\varepsilon^2\delta}$ , here 5,834 rolls. Lesson 5.7.3 shows that about 1,121 rolls do the same job.

\varepsilon = 0.1

import math import numpy as np var, eps = 35 / 12, 0.1 for n in [100, 1000, 10_000]: print(n, round(var / (n * eps**2), 4)) rng = np.random.default_rng(5722) means = rng.integers(1, 7, size=(4_000, 1000)).mean(axis=1) print(round(float(np.mean(np.abs(means - 3.5) >= eps)), 4)) print(math.ceil(var / (eps**2 * 0.05))) print(round(math.sqrt(var / 5000), 4), round(2 * math.sqrt(var / 5000), 4)) print(round(2 * math.sqrt(var / 10_000), 4), round(1 / math.sqrt(0.05) / 2, 2))

⌘/Ctrl + Enter runs the cell

One long run

The rule speaks about many repetitions of $n$ draws. A stronger version, the strong law of large numbers, speaks about one endless run: its running mean settles on $\mu$ with probability 1. That is what a single simulation shows.

Choose the die and 5,000 draws. The readout compares the miss with $\frac{\sigma}{\sqrt{n}} \approx 0.0242$ .

Press "Draw again" several times: each run wanders differently at first, and most end within $\frac{2\sigma}{\sqrt{n}} \approx 0.0483$ of 3.5. Then try the exponential, whose rare long draws make visible jumps.

Explore

Each point is the average of the first k draws. Pick a starting distribution and watch the running mean settle on the true mean as k grows; draw again for a different run.

Draws = 1000

True mean and standard deviation: μ = 3.5, σ = 1.7078
Mean of 1000 draws: 3.565, off by 0.065; a typical miss is σ/√n = 0.054

Twenty runs of 10,000 rolls settle on 3.5 inside a band that narrows like $\frac{1}{\sqrt{n}}$ . Chebyshev's guarantee, dashed, is about 2.24 times wider.

Shares settle on probabilities

A share is a mean in disguise. For an event $A$ , let the indicator $I$ be 1 when $A$ happens and 0 when not. It has mean $p = P(A)$ and variance $p(1 - p)$ (lesson 5.3.2).

The share of $n$ independent trials in which $A$ happens, written $\hat{p}_n$ , is the sample mean of $n$ indicators. The rules above apply with $\mu = p$ and $\sigma^2 = p(1 - p)$ .

Rule: A share

\begin{gathered} \mathbb{E}[\hat{p}_n] = p \\[4pt] \sigma_{\hat{p}_n} = \sqrt{\frac{p(1 - p)}{n}} \end{gathered}

The share of successes in $n$ independent trials, each a success with probability $p$ , settles on $p$ . Its standard error shrinks like $\frac{1}{\sqrt{n}}$ , and is largest at $p = \frac{1}{2}$ .

This is why the share of sevens in lesson 5.1.1 settled on $\frac{1}{6}$ , and how fast. Its standard error is 0.0373 after 100 rolls, 0.00373 after 10,000 and 0.000373 after a million.

That run's shares, 0.19, 0.1652 and 0.1665, missed $\frac{1}{6}$ by 0.0233, $-0.0015$ and $-0.0002$ . Each miss is less than one standard error: 0.63, 0.39 and 0.45 of one.

Case study 4.7.5 attached a standard error $\sqrt{\hat{p}(1 - \hat{p})/N}$ to a simulated probability. It is this rule with the unknown $p$ replaced by its estimate $\hat{p}$ : with $N = 10{,}000$ draws and $\hat{p} = 0.0293$ it is 0.00169. How often the miss stays within two standard errors is lesson 5.7.3's question.

n

import math p = 1 / 6 for n, share in [(100, 0.19), (10_000, 0.1652), (1_000_000, 0.1665)]: sd = math.sqrt(p * (1 - p) / n) print(n, round(sd, 6), round(share - p, 4), round((share - p) / sd, 2)) for N, phat in [(1000, 0.026), (10_000, 0.0293), (100_000, 0.02856), (1_000_000, 0.02892)]: print(N, round(math.sqrt(phat * (1 - phat) / N), 5))

⌘/Ctrl + Enter runs the cell

Simulation as a way to compute rests on this law. Case study 4.7.5 estimated a probability as a share, and case study 5.7.5 estimates an option price as an average.

What the law does not say

Nothing is certain at a fixed $n$ . Ten rolls can all show 6, with probability $\left(\frac{1}{6}\right)^{10} \approx 1.65 \times 10^{-8}$ , and then $\bar{X}_{10} = 6$ . The law makes large misses unlikely, never impossible.

There is no law of averages. Toss a fair coin $n$ times. The number of heads minus $\frac{n}{2}$ has standard deviation $\frac{\sqrt{n}}{2}$ : 5 after 100 tosses, 50 after 10,000, 500 after a million.

That gap grows. Only the share closes in on $\frac{1}{2}$ , because dividing by $n$ beats a gap that grows like $\sqrt{n}$ .

Nothing pushes the coin back towards even. After ten tails in a row, the next toss is still heads with probability $\frac{1}{2}$ , by independence. Expecting otherwise is the gambler's fallacy.

It needs a mean. A Student t with $\nu = 1$ degree of freedom has no mean (lesson 5.4.4). Its running mean never settles: rare, enormous draws keep jolting it.

Lesson 5.3.1's coin game with no finite expected value behaves the same.

It needs independence, or something close to it. If every "draw" is a copy of the first, $\bar{X}_n = X_1$ for every $n$ , and nothing settles. Draws that are positively correlated settle more slowly than $\frac{\sigma}{\sqrt{n}}$ says.

n

import numpy as np print((1 / 6) ** 10) rng = np.random.default_rng(5731) heads = np.cumsum(rng.integers(0, 2, size=1_000_000)) for n in [100, 10_000, 1_000_000]: print(n, int(heads[n - 1] - n / 2), round(float(heads[n - 1] / n), 4), np.sqrt(n) / 2) rng = np.random.default_rng(5724) draws = rng.standard_t(1, size=1_000_000) running = np.cumsum(draws) / np.arange(1, len(draws) + 1) print([round(float(running[n - 1]), 2) for n in [100, 1000, 10_000, 100_000, 1_000_000]])

⌘/Ctrl + Enter runs the cell

Common mistakes

The standard error is $\frac{\sigma}{\sqrt{n}}$ , not $\frac{\sigma}{n}$ . It is the variance that is divided by $n$ . A hundred times the draws give ten times the accuracy.
There is no law of averages. Past draws do not make up for each other; each new draw is independent of them.
The sum does not settle; the mean does. Heads minus $\frac{n}{2}$ grows like $\sqrt{n}$ while the share closes in.
The law promises nothing for a particular $n$ . Chebyshev's inequality gives a bound, and only a bound.
No mean, no law. The running mean of t draws with $\nu = 1$ never settles.
Copies are not independent draws. Averaging the same number $n$ times does not shrink its error.

The law of large numbers

0 of 11 correct

1.
A die is rolled 25 times. Find the variance of the sample mean, as a fraction
2.
Draws have standard deviation 5. Find the standard error of the mean of 100 of them
3.
How many times as many draws halve the standard error of a mean?
4.
Draws from an exponential with rate 2 have mean $\frac{1}{2}$ . What is the expected value of the mean of 50 of them?
5.
A fair coin is tossed 400 times. Find the standard error of the share of heads
6.
Draws have variance 4. Bound $P(|\bar{X}_{100} - \mu| \ge 0.5)$ with Chebyshev's inequality
7.
Draws have $\sigma = 1$ . How many does Chebyshev's inequality need to guarantee $P(|\bar{X}_n - \mu| \ge 0.1) \le 0.01$ ?
8.
A simulation finds a share $\hat{p} = 0.02$ from $N = 10{,}000$ draws. Find its standard error
9.
A fair coin is tossed 10,000 times. Find the standard deviation of the number of heads minus 5,000
10.
A fair coin has shown tails 9 times in a row. Find the probability that the next toss is heads
11.
One draw $X_1$ has standard deviation 3, and $\bar{X}$ averages 100 copies of that same draw. Find the standard deviation of $\bar{X}$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\mu

The law of large numbers

The sample mean is a random variable

Rule: Mean and variance of the sample mean

The law of large numbers

Rule: Law of large numbers

One long run

Shares settle on probabilities

Rule: A share

What the law does not say

Common mistakes

The law of large numbers

Summary card

Free further reading