5.6.3 Sums of independent random variables

Two dice, by counting

Roll two fair dice, $X$ and $Y$ , independently, and let $S = X + Y$ . Each of the 36 pairs has probability $\frac{1}{6} \cdot \frac{1}{6} = \frac{1}{36}$ , by independence (lesson 5.5.1). So $P(S = s)$ is the number of pairs that add up to $s$ , divided by 36.

sum $s$	2	3	4	5	6	7
pairs	1	2	3	4	5	6

From 8 to 12 the counts fall back: 5, 4, 3, 2, 1. So $P(S = 7) = \frac{6}{36} = \frac{1}{6}$ , $P(S = 4) = \frac{3}{36} = \frac{1}{12}$ , and $P(S \ge 10) = \frac{3 + 2 + 1}{36} = \frac{1}{6}$ .

Each diagonal of the grid collects the pairs with one sum. Their counts, 1 to 6 and back to 1, make the pmf of $S$ a triangle.

The same table by convolution

To get $S = s$ , the first die shows some $x$ and the second must show $s - x$ . Multiply their probabilities, by independence, and add over every $x$ . For $s = 4$ the splits are $1 + 3$ , $2 + 2$ and $3 + 1$ , so $P(S = 4) = 3 \cdot \frac{1}{36}$ .

Rule: Convolution

\begin{gathered} p_S(s) = \sum_x p_X(x)\,p_Y(s - x) \\[4pt] f_S(s) = \int_{-\infty}^{\infty} f_X(x)\,f_Y(s - x)\,dx \end{gathered}

For independent $X$ and $Y$ : add the probability of every way to split $s$ into $x$ and $s - x$ . With densities, the sum becomes an integral. The operation is called convolution.

\frac{1}{6} \approx 0.1667

import numpy as np counts = {} for x in range(1, 7): for y in range(1, 7): counts[x + y] = counts.get(x + y, 0) + 1 print(counts) p = np.full(6, 1 / 6) print([round(float(v) * 36, 6) for v in np.convolve(p, p)]) rng = np.random.default_rng(5631) S = rng.integers(1, 7, size=100_000) + rng.integers(1, 7, size=100_000) print(round(float(np.mean(S == 7)), 4), round(float(np.mean(S == 4)), 4))

⌘/Ctrl + Enter runs the cell

The MGF of a sum is the product

Convolution gets slow for long sums and needs an integral for densities. The MGF of lesson 5.6.2 turns it into a multiplication. Since $e^{t(X + Y)} = e^{tX}e^{tY}$ , the question is the mean of a product.

With $X$ and $Y$ independent, $\mathbb{E}[g(X)\,h(Y)] = \mathbb{E}[g(X)]\,\mathbb{E}[h(Y)]$ (lesson 5.5.2). The joint pmf is the product $p_X(x)\,p_Y(y)$ (lesson 5.5.1), so the double sum splits into a product of two sums.

Rule: MGF of an independent sum

\begin{gathered} M_{X + Y}(t) = M_X(t)\,M_Y(t) \\[4pt] \psi_{X + Y}(t) = \psi_X(t)\,\psi_Y(t) \end{gathered}

For independent $X$ and $Y$ , and likewise for more terms: $n$ independent copies of $X$ have MGF $M_X(t)^n$ . The characteristic functions multiply in the same way.

For two dice, $M_S(t) = \left(\frac{e^t + \cdots + e^{6t}}{6}\right)^2$ . Write $u$ for $e^t$ and multiply out:

\begin{gathered} \left(\frac{u + u^2 + \cdots + u^6}{6}\right)^2 \\[4pt] = \frac{u^2 + 2u^3 + 3u^4 + \cdots + 6u^7 + \cdots + u^{12}}{36} \end{gathered}

The coefficient of $u^s = e^{st}$ is $P(S = s)$ . Multiplying the two polynomials collects the same splits as the convolution.

t = 0.3

import math import sympy as sp counts = {s: 6 - abs(s - 7) for s in range(2, 13)} M_die = sum(math.exp(0.3 * k) for k in range(1, 7)) / 6 M_sum = sum(c * math.exp(0.3 * s) for s, c in counts.items()) / 36 print(round(M_die**2, 6), round(M_sum, 6)) u = sp.symbols("u") print(sp.expand(36 * ((u + u**2 + u**3 + u**4 + u**5 + u**6) / 6) ** 2))

⌘/Ctrl + Enter runs the cell

Sums that stay in their family

Multiply two MGFs, recognise the product, and the uniqueness rule of lesson 5.6.2 names the distribution of the sum.

Bernoullis make a binomial. A coin that shows 1 with probability $p$ has $M(t) = 1 - p + pe^t$ (lesson 5.6.2). The number of 1s in $n$ independent tosses has $M(t) = (1 - p + pe^t)^n$ . Multiplying out gives the binomial probabilities of lesson 5.2.2: with $n = 3$ and $p = 0.3$ , $(0.7 + 0.3u)^3 = 0.343 + 0.441u + 0.189u^2 + 0.027u^3$ .

Poissons make a Poisson. For a Poisson count with rate $\lambda$ (lesson 5.2.3), the series for $e^x$ (lesson 2.7.2) sums the MGF:

\begin{aligned} M(t) &= \sum_{k=0}^{\infty} e^{tk}\,\frac{e^{-\lambda}\lambda^k}{k!} = e^{-\lambda}\sum_{k=0}^{\infty} \frac{(\lambda e^t)^k}{k!} \\ &= e^{-\lambda}e^{\lambda e^t} = e^{\lambda(e^t - 1)} \end{aligned}

Two independent counts with rates $\lambda_1$ and $\lambda_2$ give $e^{\lambda_1(e^t - 1)}e^{\lambda_2(e^t - 1)} = e^{(\lambda_1 + \lambda_2)(e^t - 1)}$ : a Poisson with rate $\lambda_1 + \lambda_2$ . With rates 1 and 2, $P(S = 3) = \frac{e^{-3}\,3^3}{3!} \approx 0.224042$ , and convolution agrees.

\lambda = 2

import math import numpy as np import sympy as sp from scipy.stats import binom, poisson lam, t = 2, 0.5 series = sum(math.exp(t * k) * math.exp(-lam) * lam**k / math.factorial(k) for k in range(60)) print(round(series, 6), round(math.exp(lam * (math.exp(t) - 1)), 6)) k = np.arange(40) conv = np.convolve(poisson.pmf(k, 1), poisson.pmf(k, 2)) print([round(float(v), 6) for v in conv[:6]]) print([round(float(v), 6) for v in poisson.pmf(np.arange(6), 3)]) u = sp.symbols("u") print(sp.expand((sp.Rational(7, 10) + sp.Rational(3, 10) * u) ** 3), [round(float(v), 6) for v in binom.pmf(range(4), 3, 0.3)]) rng = np.random.default_rng(5632) S = rng.poisson(1, 100_000) + rng.poisson(2, 100_000) print(round(float(np.mean(S == 3)), 4), round(float(S.mean()), 4), round(float(S.var()), 4))

⌘/Ctrl + Enter runs the cell

Sums of independent normals

Lesson 5.4.3 stated that a sum of independent normals is normal; the MGFs show why. Multiply $e^{\mu_1 t + \sigma_1^2 t^2/2}$ by $e^{\mu_2 t + \sigma_2^2 t^2/2}$ and the exponents add:

M_{X + Y}(t) = e^{(\mu_1 + \mu_2)t + (\sigma_1^2 + \sigma_2^2)t^2/2}

That is the MGF of a normal with mean $\mu_1 + \mu_2$ and variance $\sigma_1^2 + \sigma_2^2$ .

Rule: Sums of independent normals

\begin{gathered} X \sim \mathcal{N}(\mu_1, \sigma_1^2),\; Y \sim \mathcal{N}(\mu_2, \sigma_2^2) \\[4pt] X + Y \sim \mathcal{N}(\mu_1 + \mu_2,\ \sigma_1^2 + \sigma_2^2) \end{gathered}

For independent $X$ and $Y$ . Means add and variances add; standard deviations do not. More generally $aX + bY \sim \mathcal{N}(a\mu_1 + b\mu_2,\ a^2\sigma_1^2 + b^2\sigma_2^2)$ .

An example. Take $X \sim \mathcal{N}(1, 4)$ and $Y \sim \mathcal{N}(2, 9)$ , independent. Then $X + Y \sim \mathcal{N}(3, 13)$ , with standard deviation $\sqrt{13} \approx 3.605551$ , not $2 + 3 = 5$ . So $P(X + Y > 5) = 1 - N\!\left(\frac{5 - 3}{\sqrt{13}}\right) \approx 0.28955$ ; with the wrong 5 you would get 0.344578.

Differences. $X - Y$ is $X + (-1)Y$ , so its variance is $4 + (-1)^2 \cdot 9 = 13$ again: variances add for differences too (lesson 5.5.2). Its mean is $1 - 2 = -1$ .

Many terms. $n$ independent copies of $\mathcal{N}(\mu, \sigma^2)$ add up to $\mathcal{N}(n\mu, n\sigma^2)$ . Their average, the sum divided by $n$ , is $\mathcal{N}\!\left(\mu, \frac{\sigma^2}{n}\right)$ : lesson 5.7.2 meets this $\frac{\sigma^2}{n}$ again.

Try it The first line is the exact answer, then the one with the wrong standard deviation 5. A million seeded pairs follow. rng.normal takes the standard deviation (2 and 3), not the variance. The sum's mean, standard deviation and share above 5 come out close to 3, 3.606 and 0.290, and the difference has the same spread.

import math import numpy as np from scipy.stats import norm sd = math.sqrt(4 + 9) print(round(sd, 6), round(norm.sf(5, 3, sd), 6), round(norm.sf(5, 3, 5), 6)) rng = np.random.default_rng(5633) X = rng.normal(1, 2, 1_000_000) Y = rng.normal(2, 3, 1_000_000) S = X + Y print(round(float(S.mean()), 4), round(float(S.std()), 4), round(float(np.mean(S > 5)), 4)) print(round(float((X - Y).mean()), 4), round(float((X - Y).std()), 4))

⌘/Ctrl + Enter runs the cell

A sum that leaves its family

Not every family is closed under sums. Take $X$ and $Y$ independent and exponential with rate $\lambda = 1$ , so $M(t) = \frac{1}{1 - t}$ (lesson 5.6.2). The sum has $M_S(t) = \frac{1}{(1 - t)^2}$ , which is not $\frac{\lambda'}{\lambda' - t}$ for any rate $\lambda'$ , so $S$ is not exponential.

Convolution finds its density. Both densities are 0 below 0, so $x$ runs from 0 to $s$ :

f_S(s) = \int_0^s e^{-x}\,e^{-(s - x)}\,dx = \int_0^s e^{-s}\,dx = s\,e^{-s}

This holds for $s \ge 0$ : the density starts at 0, peaks at $s = 1$ and has mean 2. A sum near 0 needs both waits to be short, which is unlikely. It is the waiting time to the second event of a Poisson stream (lesson 5.4.2), a gamma distribution.

So $P(S \le 1) = 1 - 2e^{-1} \approx 0.264241$ ; 100,000 seeded sums give 0.2638.

s

import math import numpy as np from scipy.integrate import quad for s in [0.5, 1, 1.5, 3]: conv = quad(lambda x: math.exp(-x) * math.exp(-(s - x)), 0, s)[0] print(s, round(conv, 6), round(s * math.exp(-s), 6)) print(round(1 - 2 * math.exp(-1), 6)) rng = np.random.default_rng(5634) X = rng.exponential(1, 100_000) Y = rng.exponential(1, 100_000) S = X + Y print(round(float(np.mean(S <= 1)), 4), round(float(S.mean()), 4), round(float(S.var()), 4))

⌘/Ctrl + Enter runs the cell

Where independence matters

Every rule in this lesson needs $X$ and $Y$ independent. The clearest failure is $X + X = 2X$ : one variable added to itself, the opposite of independent.

A die. $2X$ takes the values 2, 4, …, 12, each with probability $\frac{1}{6}$ . So $P(2X = 7) = 0$ , while two independent dice give 7 with probability $\frac{1}{6}$ .
The MGF. $M_{2X}(t) = M_X(2t)$ , by the rule for $aX + b$ (lesson 5.6.2), not $M_X(t)^2$ .
Exponentials. $2X$ has density $\frac{1}{2}e^{-s/2}$ (lesson 5.6.1), not $se^{-s}$ . Both have mean 2, but the variance of $2X$ is 4, against 2 for $X + Y$ .
Normals. $Z + Z = 2Z$ is $\mathcal{N}(0, 4)$ , while two independent standard normals add to $\mathcal{N}(0, 2)$ .

Two ways to add exponentials with mean 1. Independent waits rarely sum to near 0; doubling one wait often does. Both have mean 2.

Means always add, independent or not (lesson 5.3.2). Variances add when the terms are uncorrelated (lesson 5.5.2).

The whole distribution of a sum needs more: independence, or the joint distribution. Correlated normals still add to a normal when they are jointly normal (lesson 5.5.3), with $2\operatorname{Cov}(X, Y)$ in the variance.

Module 5.7 adds up many independent draws: the law of large numbers and the central limit theorem. Case study 5.6.5 adds up independent normal log returns.

Common mistakes

Add variances, not standard deviations. $\mathcal{N}(1, 4) + \mathcal{N}(2, 9)$ has standard deviation $\sqrt{13} \approx 3.61$ , not 5.
Variances add for differences too. $\operatorname{Var}(X - Y) = 13$ , not $9 - 4$ .
Multiply MGFs; do not add them. $M_X + M_Y$ is not even 1 at $t = 0$ .
$X + X$ is not two independent copies. $2X$ has $M_X(2t)$ , and a doubled die never shows 7.
Not every family is closed. Two exponentials add to $se^{-s}$ , not to an exponential.
Check independence first. Without it, the product rule and the convolution formula do not apply.

Sums of independent random variables

0 of 12 correct

1.
Two fair dice are rolled. Find $P(\text{sum} = 4)$
2.
Two fair dice are rolled. Find $P(\text{sum} \ge 10)$
3.
A fair coin $X$ is 0 or 1, and a fair die $Y$ is rolled independently. Find $P(X + Y = 6)$
4.
$S$ counts the 1s in 10 independent tosses of a coin with $p = 0.3$ . Find $P(S = 0)$ (a power is fine)
5.
Write the MGF of the number of 1s in 3 independent tosses with $p = 0.2$
6.
$X$ and $Y$ are independent Poisson counts with rates 2 and 3. Find $P(X + Y = 0)$ exactly
7.
$X \sim \mathcal{N}(1, 4)$ and $Y \sim \mathcal{N}(2, 9)$ are independent. Find the standard deviation of $X + Y$ (exactly)
8.
With the same $X$ and $Y$ , find $\operatorname{Var}(X - Y)$
9.
$X_1, \ldots, X_4$ are independent, each $\mathcal{N}(5, 1)$ . Find the variance of their average
10.
$X$ and $Y$ are independent with $M_X(t)M_Y(t) = e^{4t + 8t^2}$ , and $X \sim \mathcal{N}(1, 6)$ . Find $\operatorname{Var}(Y)$
11.
$X$ and $Y$ are independent exponentials with rate 1. Find the density of $X + Y$ at $s = 1$ , exactly
12.
$X$ is one fair die. Find $P(2X = 7)$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

X

Sums of independent random variables

Two dice, by counting

The same table by convolution

Rule: Convolution

The MGF of a sum is the product

Rule: MGF of an independent sum

Sums that stay in their family

Sums of independent normals

Rule: Sums of independent normals

A sum that leaves its family

Where independence matters

Common mistakes

Sums of independent random variables

Summary card

Free further reading