5.3.2 Variance, and linearity of expectation

Spread around the mean

Two coin games have the same mean. In game A you win or lose $1; in game B you win or lose $10. Both have expected winnings of 0, but B is far riskier.

The mean says nothing about how far the outcomes land from it.

The distance of a value from the mean $\mu = \mathbb{E}[X]$ is its deviation, $x - \mu$ . Deviations above and below the mean cancel, so their average is always 0. Squaring them first stops the cancelling.

Rule: Variance and standard deviation

\begin{gathered} \operatorname{Var}(X) = \mathbb{E}\big[(X - \mu)^2\big] \\[4pt] \sigma_X = \sqrt{\operatorname{Var}(X)} \end{gathered}

The variance is the expected squared deviation, found by LOTUS (lesson 5.3.1) with $g(x) = (x - \mu)^2$ . It is in squared units, such as dollars squared. The standard deviation $\sigma_X$ (sigma) is back in the units of $X$ .

Game A has variance $\frac{1}{2} \cdot 1^2 + \frac{1}{2} \cdot (-1)^2 = 1$ and standard deviation $1. Game B has variance 100 and standard deviation $10: ten times the spread.

For a fair die, $\mu = 3.5$ and each face has probability $\frac{1}{6}$ :

$x$	1	2	3	4	5	6
$x - \mu$	$-2.5$	$-1.5$	$-0.5$	0.5	1.5	2.5
$(x - \mu)^2$	6.25	2.25	0.25	0.25	2.25	6.25

The squares add up to 17.5, so the variance is $\frac{17.5}{6} = \frac{35}{12} \approx 2.9167$ . The standard deviation is $\sqrt{35/12} \approx 1.7078$ dots.

Two distributions with mean 3. The bracket under each runs from one standard deviation below the mean to one above: 1.22 for the binomial, 2 for the flat one.

Three cases follow from the rule.

Never negative. A variance is a sum of squares times probabilities.
Zero only for a constant. If $X$ always takes the same value, every deviation is 0.
Bernoulli( $p$ ): $\operatorname{Var}(X) = p(1 - p)$ . It is largest, 0.25, at $p = \frac{1}{2}$ , and 0 at $p = 0$ or 1, where the outcome is certain.

\frac{35}{12}

import math from fractions import Fraction def variance(values, probs): mu = sum(x * p for x, p in zip(values, probs)) return sum((x - mu) ** 2 * p for x, p in zip(values, probs)) die = variance([1, 2, 3, 4, 5, 6], [Fraction(1, 6)] * 6) print(die, round(math.sqrt(die), 4)) for p in [0, 0.1, 0.5, 0.9, 1]: print(p, round(variance([0, 1], [1 - p, p]), 10))

⌘/Ctrl + Enter runs the cell

A shortcut: the mean of the square minus the square of the mean

Expanding the square gives a second formula, often quicker by hand.

Rule: Variance shortcut

\operatorname{Var}(X) = \mathbb{E}[X^2] - \mu^2

The mean of the square minus the square of the mean: exactly the gap of lesson 5.3.1. So $\mathbb{E}[X^2] \ge (\mathbb{E}[X])^2$ , with equality only for a constant.

Take $X$ equal to 0, 1 or 2 with probabilities 0.5, 0.3 and 0.2, so $\mu = 0.7$ . Both formulas give 0.61:

Definition: $0.49 \times 0.5 + 0.09 \times 0.3 + 1.69 \times 0.2 = 0.245 + 0.027 + 0.338 = 0.61$ .
Shortcut: $\mathbb{E}[X^2] = 0 + 0.3 + 0.8 = 1.1$ , and $1.1 - 0.7^2 = 0.61$ .

For the die, $\mathbb{E}[X^2] = \frac{91}{6}$ (lesson 5.3.1), so the shortcut gives $\frac{91}{6} - \frac{49}{4} = \frac{35}{12}$ again.

On a computer the shortcut has a trap. Add $10^8$ to every face: the spread is still $\frac{35}{12}$ , but $\mathbb{E}[X^2]$ is about $10^{16}$ . A computer keeps about 16 digits (lesson 2.8.1), so subtracting two such nearly equal numbers leaves only rounding error.

10^8

import numpy as np x = np.array([0, 1, 2]) p = np.array([0.5, 0.3, 0.2]) mu = x @ p print(round(((x - mu) ** 2) @ p, 10), round((x**2) @ p - mu**2, 10)) big = 1e8 + np.arange(1, 7) shortcut = (big**2).mean() - big.mean() ** 2 print(round(np.var(big), 6), abs(shortcut - 35 / 12) > 0.5)

⌘/Ctrl + Enter runs the cell

Shifting and scaling

Change the units, or add a fee: many random variables are $aX + b$ for some $X$ whose mean and variance you know.

Rule: Linear change

\begin{gathered} \mathbb{E}[aX + b] = a\,\mathbb{E}[X] + b \\[4pt] \operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X) \\[4pt] \sigma_{aX + b} = |a|\,\sigma_X \end{gathered}

Adding $b$ moves every value and the mean by $b$ , so the deviations do not change. Multiplying by $a$ multiplies each deviation by $a$ and each squared deviation by $a^2$ .

A fee and a prize. Pay $20 to roll a die and win $10 for each dot, so you gain $Y = 10X - 20$ dollars. Then $\mathbb{E}[Y] = 10 \times 3.5 - 20 = 15$ dollars. The variance is $100 \times \frac{35}{12} \approx 291.67$ dollars squared, and the standard deviation is $10 \times 1.7078 \approx 17.08$ dollars.

A negative $a$ . The other side of a bet, $-X$ , has the same spread: $\operatorname{Var}(-X) = (-1)^2\operatorname{Var}(X)$ . That is why $\sigma$ takes $|a|$ .
Units. In cents, $100X$ has a mean and standard deviation 100 times larger, and a variance 10,000 times larger.

Standardising

Subtract the mean and divide by the standard deviation:

Z = \frac{X - \mu}{\sigma_X}

This is $aX + b$ with $a = \frac{1}{\sigma_X}$ and $b = -\frac{\mu}{\sigma_X}$ , so $\mathbb{E}[Z] = 0$ and $\operatorname{Var}(Z) = 1$ . $Z$ counts standard deviations from the mean and has no units.

For the die, $Z$ runs from $-1.4639$ to 1.4639. Lesson 5.3.3 builds on $Z$ .

Y = 10X - 20

import numpy as np x = np.arange(1, 7) p = np.full(6, 1 / 6) def mean_var(values): mu = values @ p return round(float(mu), 4) + 0.0, round(float(((values - mu) ** 2) @ p), 4) print(mean_var(x)) print(mean_var(10 * x - 20), round(100 * 35 / 12, 4)) print(mean_var(-x)) z = (x - 3.5) / np.sqrt(35 / 12) print(np.round(z, 4), mean_var(z))

⌘/Ctrl + Enter runs the cell

Linearity of expectation

The mean of a sum is the sum of the means. This is the most useful rule about expected values, because it needs nothing about how the variables relate.

Rule: Linearity of expectation

\begin{gathered} \mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\[4pt] \mathbb{E}\Big[\sum_i a_i X_i\Big] = \sum_i a_i\,\mathbb{E}[X_i] \end{gathered}

It holds for any random variables on the same experiment, independent or not, and with any constants $a_i$ .

Two dice. The total of two dice has mean $3.5 + 3.5 = 7$ , with no need for the table of 36 totals.

The top and the bottom. Opposite faces of a die add up to 7, so the face underneath is $7 - X$ . The two faces are as dependent as can be, yet $\mathbb{E}[X + (7 - X)] = 3.5 + 3.5 = 7$ still.

Counting with indicators

An indicator $I_A$ is 1 if the event $A$ happens and 0 if not. It is a Bernoulli variable, so $\mathbb{E}[I_A] = P(A)$ . A count is a sum of indicators, so its mean is a sum of probabilities.

Put 3 letters into 3 envelopes in a random order. How many land in their own envelope, on average? List the 6 orders, writing which letter each envelope gets:

order	123	132	213	231	312	321
letters in the right envelope	3	1	1	0	0	1

The average is $\frac{6}{6} = 1$ . With indicators: each letter is in its own envelope with probability $\frac{1}{3}$ , so the mean count is $3 \times \frac{1}{3} = 1$ .

The indicators are not independent: if two letters are right, so is the third. Linearity does not care. With $n$ letters the answer is $n \times \frac{1}{n} = 1$ , for every $n$ .

n

from itertools import permutations import numpy as np for n in [3, 4, 5, 6]: counts = [sum(1 for i, j in enumerate(order) if i == j) for order in permutations(range(n))] print(n, len(counts), np.mean(counts)) rng = np.random.default_rng(532) shuffles = np.array([rng.permutation(52) for _ in range(100_000)]) print((shuffles == np.arange(52)).sum(axis=1).mean())

⌘/Ctrl + Enter runs the cell

Variances add for independent variables

Variances do not always add. They do when the variables are independent: knowing one changes no probability about the other (lesson 5.1.3).

Rule: Variance of a sum

\begin{gathered} \operatorname{Var}(X + Y) \\[4pt] = \operatorname{Var}(X) + \operatorname{Var}(Y) \end{gathered}

True when $X$ and $Y$ are independent. Without independence it can fail; lesson 5.5.2 gives the general rule, with an extra covariance term.

sum	independent?	variance
two separate dice	yes	$\frac{35}{12} + \frac{35}{12} = \frac{35}{6} \approx 5.8333$
top face plus bottom face	no	0, since the total is always 7
$X + X = 2X$	no	$4 \times \frac{35}{12} = \frac{35}{3} \approx 11.67$

The last row is a common slip. A variable is not independent of itself, so $\operatorname{Var}(X + X)$ is $\operatorname{Var}(2X) = 4\operatorname{Var}(X)$ , by the scaling rule.

\frac{35}{6}

import numpy as np faces = np.arange(1, 7) totals = (faces[:, None] + faces[None, :]).ravel() print(len(totals), totals.mean(), round(totals.var(), 4), round(35 / 6, 4)) print((faces + (7 - faces)).var()) print(round((2 * faces).var(), 4), round(35 / 3, 4))

⌘/Ctrl + Enter runs the cell

The binomial's mean and variance

A binomial variable $X \sim \operatorname{Bin}(n, p)$ counts the successes in $n$ independent trials, each a success with probability $p$ (lesson 5.2.2). It is a sum of indicators, $X = I_1 + I_2 + \cdots + I_n$ , where $I_j$ is 1 when trial $j$ succeeds. Each $I_j$ is Bernoulli( $p$ ), with mean $p$ and variance $p(1 - p)$ .

Rule: Binomial mean and variance

\begin{gathered} \mathbb{E}[X] = np \\[4pt] \operatorname{Var}(X) = np(1 - p) \end{gathered}

The mean comes from linearity alone. The variance needs the trials to be independent, so that the $n$ Bernoulli variances add.

Count the sixes in 10 rolls of a die: $n = 10$ and $p = \frac{1}{6}$ . The mean is $\frac{10}{6} = \frac{5}{3} \approx 1.6667$ and the variance is $10 \times \frac{1}{6} \times \frac{5}{6} = \frac{25}{18} \approx 1.3889$ . The standard deviation is about 1.1785.

The same rules give the whole of module 5.2:

distribution	$\mathbb{E}[X]$	$\operatorname{Var}(X)$
Bernoulli( $p$ )	$p$	$p(1 - p)$
$\operatorname{Bin}(n, p)$	$np$	$np(1 - p)$
$\operatorname{Geom}(p)$	$\frac{1}{p}$	$\frac{1 - p}{p^2}$
$\operatorname{Poisson}(\lambda)$	$\lambda$	$\lambda$

The geometric and Poisson variances are stated, and checked below. Waiting for a six has mean 6 and variance 30, a standard deviation of about 5.48 rolls. A Poisson count has its variance equal to its mean.

p = \frac{1}{6}

import numpy as np from scipy import stats k = np.arange(0, 11) pmf = stats.binom.pmf(k, 10, 1 / 6) mu = k @ pmf print(round(mu, 4), round(((k - mu) ** 2) @ pmf, 4), [round(float(v), 4) for v in stats.binom(10, 1 / 6).stats(moments="mv")]) rng = np.random.default_rng(532) counts = rng.binomial(10, 1 / 6, size=100_000) print(counts.mean(), round(counts.var(), 5)) k = np.arange(1, 400) pmf = stats.geom.pmf(k, 1 / 6) print(round(k @ pmf, 6), round(((k - 6) ** 2) @ pmf, 6)) k = np.arange(0, 60) pmf = stats.poisson.pmf(k, 2.5) print(round(k @ pmf, 6), round(((k - 2.5) ** 2) @ pmf, 6))

⌘/Ctrl + Enter runs the cell

Common mistakes

Means always add; variances add only for independent variables. The top and bottom faces of a die have a total with variance 0, not $\frac{35}{6}$ .
In $\operatorname{Var}(aX + b)$ the shift drops out and $a$ is squared. $\operatorname{Var}(3X + 5) = 9\operatorname{Var}(X)$ .
A variance is never negative. $\operatorname{Var}(-X) = \operatorname{Var}(X)$ , and $\sigma_{aX} = |a|\,\sigma_X$ .
$X + X$ is not two independent copies. $\operatorname{Var}(X + X) = 4\operatorname{Var}(X)$ , not $2\operatorname{Var}(X)$ .
Variance is in squared units. Compare spreads, and quote them, with the standard deviation.
The shortcut needs $\mathbb{E}[X^2]$ , not $(\mathbb{E}[X])^2$ . On a computer, use the definition when the mean is large compared with the spread.

Variance and linearity

0 of 13 correct

1.
$X$ is 0, 1 or 2 with probabilities 0.5, 0.3 and 0.2, so $\mathbb{E}[X] = 0.7$ . Find $\operatorname{Var}(X)$
2.
A coin game wins or loses 10 dollars with equal chances. Find the standard deviation of the winnings, in dollars
3.
$\mathbb{E}[X] = 2$ and $\mathbb{E}[X^2] = 7$ . Find $\operatorname{Var}(X)$
4.
Find the variance of a fair die roll, as a fraction
5.
Find the variance of a Bernoulli variable with $p = 0.1$
6.
$\operatorname{Var}(X) = 4$ . Find $\operatorname{Var}(3X + 5)$
7.
$X$ has standard deviation 3. Find the standard deviation of $1 - 2X$
8.
$X$ has mean 10 and standard deviation 2. Standardise the value $x = 13$
9.
Find the expected total of three dice, as a fraction
10.
10 letters go into 10 envelopes in a random order. How many land in their own envelope, on average?
11.
A fair coin is tossed 100 times. Find the standard deviation of the number of heads
12.
Two dice are rolled independently. Find the variance of their total, as a fraction
13.
$\operatorname{Var}(X) = 2$ . Find $\operatorname{Var}(X + X)$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\operatorname{Var}(X) = \mathbb{E}[(X - \mu)^2] = \mathbb{E}[X^2] - \mu^2

Variance, and linearity of expectation

Spread around the mean

Rule: Variance and standard deviation

A shortcut: the mean of the square minus the square of the mean

Rule: Variance shortcut

Shifting and scaling

Rule: Linear change

Standardising

Linearity of expectation

Rule: Linearity of expectation

Counting with indicators

Variances add for independent variables

Rule: Variance of a sum

The binomial's mean and variance

Rule: Binomial mean and variance

Common mistakes

Variance and linearity

Summary card

Free further reading