5.6.2 Moment generating and characteristic functions

In this lesson

Moments $\mathbb{E}[X^k]$ , and what the first four tell you
The moment generating function $M_X(t) = \mathbb{E}[e^{tX}]$ : one function that holds every moment
Its derivatives at 0 are the moments, read off the series for $e^x$ (lesson 2.7.2)
Where an MGF exists, and where it does not
The normal's MGF, $e^{\mu t + \sigma^2 t^2/2}$ , and the rule for $aX + b$
The characteristic function $\mathbb{E}[e^{itX}]$ , built from Euler's formula: it always exists
Either function pins down the distribution

You need expectation, LOTUS and variance (lessons 5.3.1 and 5.3.2), and the exponential and normal distributions (lessons 5.4.2 and 5.4.3). You also need the series for $e^x$ and $\frac{1}{1 - x}$ (lesson 2.7.2), Euler's formula (lesson 1.7.3) and improper integrals (lesson 2.6.3).

Moments

The $k$ -th moment of a random variable $X$ is $\mathbb{E}[X^k]$ , the average of its $k$ -th power. The first is the mean $\mu$ . With the second you get the variance, $\operatorname{Var}(X) = \mathbb{E}[X^2] - \mu^2$ (lesson 5.3.2), and the third and fourth give the skewness and kurtosis of lesson 5.3.3.

For a fair die $X$ , LOTUS (lesson 5.3.1) gives $\mathbb{E}[X] = \frac{1 + 2 + \cdots + 6}{6} = 3.5$ and $\mathbb{E}[X^2] = \frac{1 + 4 + \cdots + 36}{6} = \frac{91}{6} \approx 15.1667$ . So $\operatorname{Var}(X) = \frac{91}{6} - 3.5^2 = \frac{35}{12} \approx 2.9167$ .

Each moment is a separate sum or integral. This lesson packs all of them into one function.

The moment generating function

Rule: Moment generating function

M_X(t) = \mathbb{E}\big[e^{tX}\big]

For each number $t$ , the average of $e^{tX}$ , by LOTUS. At $t = 0$ it is always $\mathbb{E}[1] = 1$ . The product $tX$ must have no units, so $t$ is measured per unit of $X$ .

A coin. A Bernoulli variable is 1 with probability $p$ and 0 otherwise (lesson 5.2.2). So $M(t) = (1 - p)e^{0} + pe^{t} = 1 - p + pe^t$ .

A die. Each face $k$ adds $\frac{1}{6}e^{tk}$ , so $M(t) = \frac{e^t + e^{2t} + \cdots + e^{6t}}{6}$ . At $t = 0.3$ it is 3.247172, and 100,000 seeded rolls give an average of $e^{0.3X}$ of 3.2544. It is not $e^{0.3 \times 3.5} \approx 2.857651$ : the average of $e^{tX}$ is not $e$ to the average (lesson 5.3.1).

An exponential. With rate $\lambda$ , LOTUS gives an integral (lesson 2.6.3):

\begin{aligned} M(t) &= \int_0^\infty e^{tx}\,\lambda e^{-\lambda x}\,dx \\ &= \lambda\int_0^\infty e^{-(\lambda - t)x}\,dx = \frac{\lambda}{\lambda - t} \end{aligned}

This holds only for $t < \lambda$ . At $t \ge \lambda$ the exponent no longer sinks, and the integral diverges. With $\lambda = 2$ and $t = 2.5$ , the integral up to 10, 20 and 40 is 589.7, 88101.9 and 1940660777.6.

\mathbb{E}[X^2]

import math import numpy as np from scipy.integrate import quad def M(t): return sum(math.exp(t * k) for k in range(1, 7)) / 6 print(sum(range(1, 7)) / 6, round(sum(k * k for k in range(1, 7)) / 6, 4)) print(M(0), round(M(0.3), 6), round(math.exp(0.3 * 3.5), 6)) rng = np.random.default_rng(5621) X = rng.integers(1, 7, size=100_000) print(round(float(np.exp(0.3 * X).mean()), 4)) lam = 2 for t in [0.5, 1, 1.5]: print(t, round(quad(lambda x: lam * math.exp((t - lam) * x), 0, math.inf)[0], 6), round(lam / (lam - t), 6)) for b in [10, 20, 40]: print(b, round(quad(lambda x: lam * math.exp((2.5 - lam) * x), 0, b)[0], 1))

⌘/Ctrl + Enter runs the cell

Moments from the derivatives at 0

The series for $e^x$ (lesson 2.7.2), with $x = tX$ , is $e^{tX} = 1 + tX + \frac{t^2X^2}{2!} + \frac{t^3X^3}{3!} + \cdots$ . Average each term and the moments appear as coefficients:

\begin{aligned} M_X(t) = 1 &+ t\,\mathbb{E}[X] + \frac{t^2}{2!}\,\mathbb{E}[X^2] \\ &+ \frac{t^3}{3!}\,\mathbb{E}[X^3] + \cdots \end{aligned}

This is a Taylor series about 0, so its coefficients are the derivatives at 0 divided by $k!$ (lesson 2.7.2). Differentiating once and setting $t = 0$ leaves $\mathbb{E}[X]$ ; twice leaves $\mathbb{E}[X^2]$ .

Rule: Moments from the MGF

\begin{gathered} M_X^{(k)}(0) = \mathbb{E}[X^k] \\[4pt] M_X'(0) = \mu, \quad M_X''(0) = \mathbb{E}[X^2] \end{gathered}

The $k$ -th derivative at $t = 0$ is the $k$ -th moment, when the MGF exists on an interval around 0. On the graph, the slope of $M$ at 0 is the mean.

The die again. $M'(t) = \frac{e^t + 2e^{2t} + \cdots + 6e^{6t}}{6}$ , so $M'(0) = 3.5$ . In the same way $M''(0) = \frac{1 + 4 + \cdots + 36}{6} = \frac{91}{6}$ , and $M'''(0) = \frac{147}{2}$ .

The exponential. Here the series comes for free. $\frac{\lambda}{\lambda - t} = \frac{1}{1 - t/\lambda}$ is the geometric series of lesson 2.7.2 with ratio $\frac{t}{\lambda}$ :

M(t) = 1 + \frac{t}{\lambda} + \frac{t^2}{\lambda^2} + \frac{t^3}{\lambda^3} + \cdots

Matching each coefficient with $\frac{\mathbb{E}[X^k]}{k!}$ gives $\mathbb{E}[X^k] = \frac{k!}{\lambda^k}$ . So the mean is $\frac{1}{\lambda}$ and the variance $\frac{2}{\lambda^2} - \frac{1}{\lambda^2} = \frac{1}{\lambda^2}$ , as in lesson 5.4.2. With $\lambda = 2$ , $\mathbb{E}[X^3] = \frac{6}{8} = 0.75$ ; a million seeded draws give 0.7451.

The series works only for $\lvert t \rvert < \lambda$ , its radius of convergence (lesson 2.7.2). Its right edge, $t = \lambda$ , is where the MGF stops existing.

Every MGF passes through 1 at $t = 0$ , with slope equal to the mean. The exponential's MGF blows up at $t = \lambda = 2$ , and beyond that its integral diverges.

\lambda = 2

import math import numpy as np import sympy as sp t = sp.symbols("t") M = sum(sp.exp(k * t) for k in range(1, 7)) / 6 print([sp.diff(M, t, k).subs(t, 0) for k in [1, 2, 3]]) def M_num(s): return sum(math.exp(s * k) for k in range(1, 7)) / 6 h = 1e-4 print(round((M_num(h) - M_num(-h)) / (2 * h), 6), round((M_num(h) - 2 * M_num(0) + M_num(-h)) / h**2, 4)) print(sp.series(2 / (2 - t), t, 0, 4)) rng = np.random.default_rng(5622) X = rng.exponential(1 / 2, size=1_000_000) print(round(float(X.mean()), 4), round(float((X**2).mean()), 4), round(float((X**3).mean()), 4))

⌘/Ctrl + Enter runs the cell

The normal's MGF

For a standard normal $Z$ , with density $\varphi$ , the MGF is $\int e^{tz}\varphi(z)\,dz$ . Complete the square in the exponent, as in case study 2.6.5:

tz - \frac{z^2}{2} = -\frac{(z - t)^2}{2} + \frac{t^2}{2}

So $e^{tz}\varphi(z) = e^{t^2/2}\,\varphi(z - t)$ : a bell shifted right by $t$ , scaled by $e^{t^2/2}$ . The shifted bell has area 1, so $M_Z(t) = e^{t^2/2}$ , for every $t$ .

Moving and stretching follows from the definition. $\mathbb{E}[e^{t(aX + b)}] = e^{bt}\,\mathbb{E}[e^{(at)X}]$ , so $M_{aX + b}(t) = e^{bt}M_X(at)$ . A normal $X = \mu + \sigma Z$ (lesson 5.6.1) therefore has $M_X(t) = e^{\mu t}e^{\sigma^2 t^2/2}$ .

Rule: MGF of the normal

\begin{gathered} Z \sim \mathcal{N}(0, 1){:}\quad M_Z(t) = e^{t^2/2} \\[6pt] X \sim \mathcal{N}(\mu, \sigma^2){:} \\[2pt] M_X(t) = e^{\mu t + \sigma^2 t^2/2} \end{gathered}

It exists for every $t$ . The general rule behind the second line is $M_{aX + b}(t) = e^{bt}M_X(at)$ .

Moments of $Z$ . Put $\frac{t^2}{2}$ into the series for $e^x$ : $e^{t^2/2} = 1 + \frac{t^2}{2} + \frac{t^4}{8} + \frac{t^6}{48} + \cdots$ . There are no odd powers, so the odd moments are 0, as the bell's symmetry says.

The coefficient of $t^2$ is $\frac{\mathbb{E}[Z^2]}{2!} = \frac{1}{2}$ , so $\mathbb{E}[Z^2] = 1$ . That of $t^4$ is $\frac{\mathbb{E}[Z^4]}{4!} = \frac{1}{8}$ , so $\mathbb{E}[Z^4] = 3$ : the normal's kurtosis of 3 in lesson 5.3.3. In the same way $\mathbb{E}[Z^6] = \frac{720}{48} = 15$ .

An example. For $X \sim \mathcal{N}(1, 4)$ , $M(t) = e^{t + 2t^2}$ . Then $M'(0) = 1$ and $M''(0) = 5$ , so the variance is $5 - 1^2 = 4$ .

At $t = 1$ . $M_X(1) = \mathbb{E}[e^X] = e^{\mu + \sigma^2/2}$ , the mean of the lognormal $e^X$ of lesson 5.4.4. Case study 5.6.5 uses it for a stock price.

e^{tz}\varphi(z)

import math import numpy as np import sympy as sp from scipy.integrate import quad for s in [0.5, 1, 2]: val = quad(lambda z: math.exp(s * z - z * z / 2) / math.sqrt(2 * math.pi), -math.inf, math.inf)[0] print(s, round(val, 6), round(math.exp(s**2 / 2), 6)) rng = np.random.default_rng(5623) Z = rng.standard_normal(1_000_000) print(round(float(np.exp(0.5 * Z).mean()), 4)) t = sp.symbols("t") print(sp.series(sp.exp(t**2 / 2), t, 0, 8)) print([sp.diff(sp.exp(t**2 / 2), t, k).subs(t, 0) for k in range(1, 7)]) M = sp.exp(t + 2 * t**2) print(sp.diff(M, t).subs(t, 0), sp.diff(M, t, 2).subs(t, 0))

⌘/Ctrl + Enter runs the cell

The characteristic function

Some distributions have no MGF at all. The Student t with 1 degree of freedom, the Cauchy distribution of lesson 5.4.4, has density $\frac{1}{\pi(1 + x^2)}$ .

Its tail sinks only like $\frac{1}{x^2}$ , and any $e^{tx}$ with $t > 0$ outgrows that. Over $[-b, b]$ , the integral for $\mathbb{E}[e^{0.1X}]$ is 0.9646 at $b = 10$ , 10.21 at $b = 100$ and about $8.7 \times 10^{37}$ at $b = 1000$ .

The fix is an imaginary $t$ . By Euler's formula (lesson 1.7.3), $e^{itx} = \cos(tx) + i\sin(tx)$ is the point at angle $tx$ on the unit circle. Its size is always 1, so its average always exists.

Rule: Characteristic function

\begin{gathered} \psi_X(t) = \mathbb{E}\big[e^{itX}\big] \\[4pt] = \mathbb{E}[\cos tX] + i\,\mathbb{E}[\sin tX] \end{gathered}

It exists for every distribution and every real $t$ , with $\psi_X(0) = 1$ and $\lvert \psi_X(t) \rvert \le 1$ . Many books write $\phi_X$ ; this course keeps $\varphi$ for the normal density.

For the die, $\psi(t)$ averages six points: $e^{it}, e^{2it}, \ldots, e^{6it}$ , at angles $t, 2t, \ldots, 6t$ . Their average is their balance point, which lies inside the circle. At $t = 0.5$ it is $-0.119777 + 0.661214i$ , of size 0.671975; 100,000 seeded rolls give $-0.1182 + 0.6611i$ .

The characteristic function averages points on the unit circle, weighted by their probabilities. Two distributions with the same mean and variance have characteristic functions that agree near 0 and part further out.

Symmetric distributions give real values. A coin that is $+1$ or $-1$ with probability $\frac{1}{2}$ each has $\psi(t) = \frac{e^{it} + e^{-it}}{2} = \cos t$ : the sine parts cancel. The standard normal is symmetric too: $\psi_Z(t) = e^{-t^2/2}$ , which is $M_Z$ with $it$ in place of $t$ . It is stated here and checked by quad.

The rule for $aX + b$ carries over, so $\mathcal{N}(\mu, \sigma^2)$ has $\psi(t) = e^{i\mu t - \sigma^2 t^2/2}$ .

No MGF, but a characteristic function. The Student t with 1 degree of freedom has $\psi(t) = e^{-\lvert t \rvert}$ , with a corner at 0. That corner matches its missing mean, since $\psi'(0) = i\,\mathbb{E}[X]$ when the mean exists. In general $\psi^{(k)}(0) = i^k\,\mathbb{E}[X^k]$ .

e^{i\theta}

import cmath import math import numpy as np from scipy.integrate import quad from scipy.stats import norm, t as student_t def psi(t): return sum(cmath.exp(1j * t * k) for k in range(1, 7)) / 6 v = psi(0.5) print(round(v.real, 6), round(v.imag, 6), round(abs(v), 6), round(abs(psi(math.pi / 3)), 6)) rng = np.random.default_rng(5624) s = np.exp(1j * 0.5 * rng.integers(1, 7, size=100_000)).mean() print(round(float(s.real), 4), round(float(s.imag), 4)) for t in [0.5, 1, 2]: re = quad(lambda z: math.cos(t * z) * norm.pdf(z), -math.inf, math.inf)[0] im = quad(lambda z: math.sin(t * z) * norm.pdf(z), -math.inf, math.inf)[0] cauchy = 2 * quad(student_t(1).pdf, 0, np.inf, weight="cos", wvar=t)[0] print(t, round(re, 6), round(im, 6) + 0.0, round(math.exp(-t**2 / 2), 6), round(cauchy, 6), round(math.exp(-t), 6)) for b in [10, 100, 1000]: val = quad(lambda x: math.exp(0.1 * x) * student_t(1).pdf(x), -b, b, limit=200)[0] print(b, f"{val:.4g}")

⌘/Ctrl + Enter runs the cell

Either function pins down the distribution

The moments come out of the MGF, but the MGF carries more: the whole distribution. This lesson states the fact without proof.

Rule: Uniqueness

\begin{gathered} M_X = M_Y \text{ near } 0, \text{ or } \psi_X = \psi_Y \\[4pt] \Rightarrow\; \text{the same distribution} \end{gathered}

Two random variables whose MGFs agree on an interval around 0 have the same distribution. Characteristic functions work the same way, and they exist for every distribution.

So a recognised MGF names the distribution. $M(t) = e^{3t + 8t^2}$ is the MGF of $\mathcal{N}(3, 16)$ , because $\frac{\sigma^2}{2} = 8$ . $M(t) = 0.2 + 0.8e^t$ is a coin with $p = 0.8$ , and $\frac{e^t + e^{2t} + e^{3t}}{3}$ is equally likely to be 1, 2 or 3.

A mean and a variance are not enough. The standard normal and the uniform on $[-\sqrt{3}, \sqrt{3}]$ both have mean 0 and variance 1. Their characteristic functions, $e^{-t^2/2}$ and $\frac{\sin(\sqrt{3}t)}{\sqrt{3}t}$ , share $1 - \frac{t^2}{2}$ and differ from the $t^4$ term on.

t

import math import sympy as sp a = math.sqrt(3) for t in [0.5, 1, 2, 3]: print(t, round(math.exp(-t**2 / 2), 6), round(math.sin(a * t) / (a * t), 6)) t = sp.symbols("t") print(sp.series(sp.exp(-t**2 / 2), t, 0, 6)) print(sp.series(sp.sin(sp.sqrt(3) * t) / (sp.sqrt(3) * t), t, 0, 6))

⌘/Ctrl + Enter runs the cell

Lesson 5.6.3 uses MGFs to find the distribution of a sum. Case study 5.6.5 uses the normal's MGF for the mean of a stock price. The Financial Computing with Python course prices options from characteristic functions by Fourier methods.

Common mistakes

$M(t)$ is $\mathbb{E}[e^{tX}]$ , not $e^{t\,\mathbb{E}[X]}$ . For the die at $t = 0.3$ they are 3.247172 and 2.857651.
Differentiate, then set $t = 0$ . $M(1) = \mathbb{E}[e^X]$ is not a moment.
$M''(0)$ is $\mathbb{E}[X^2]$ , not the variance. Subtract the squared mean: $\frac{91}{6} - 3.5^2 = \frac{35}{12}$ for the die.
An MGF may exist only near 0, or not at all. The exponential's needs $t < \lambda$ ; the Student t has none. The characteristic function always exists.
Move and stretch inside. $M_{aX + b}(t) = e^{bt}M_X(at)$ , not $aM_X(t) + b$ .
The normal's characteristic function sinks. It is $e^{-t^2/2}$ , of size at most 1, while its MGF $e^{t^2/2}$ grows.

Moment generating and characteristic functions

0 of 12 correct

1.
$X$ is 1 with probability 0.3 and 0 otherwise. Find $M_X(\ln 2)$
2.
$X$ is 1 or 2 with probability $\frac{1}{2}$ each. Write $M_X(t)$
3.
$M(t) = \frac{1 + e^t + e^{2t}}{3}$ . Find the variance of $X$
4.
$M(t) = \frac{1}{1 - 2t}$ for $t$ below $\frac{1}{2}$ . Find $\mathbb{E}[X^2]$
5.
The exponential with rate $\lambda = 3$ has $M(t) = \frac{3}{3 - t}$ . From which value of $t$ on does its MGF fail to exist?
6.
$M(t) = e^{2t + 4.5t^2}$ . What is the variance of $X$ ?
7.
$Z$ is standard normal and $Y = 3Z + 1$ . Write $M_Y(t)$
8.
$e^{t^2/2} = 1 + \frac{t^2}{2} + \frac{t^4}{8} + \cdots$ . Find $\mathbb{E}[Z^4]$
9.
$X \sim \mathcal{N}(0.1, 0.04)$ . Find $\mathbb{E}[e^X]$ exactly
10.
$X$ is $+1$ or $-1$ with probability $\frac{1}{2}$ each. Find $\psi_X(\pi)$
11.
$Z$ is standard normal. Find $\psi_Z(2)$ exactly
12.
$X$ always equals 3. Find $\psi_X\left(\frac{\pi}{6}\right)$ (type i for $i$ )

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\mathbb{E}[X^k]

Moment generating and characteristic functions

Moments

The moment generating function

Rule: Moment generating function

Moments from the derivatives at 0

Rule: Moments from the MGF

The normal's MGF

Rule: MGF of the normal

The characteristic function

Rule: Characteristic function

Either function pins down the distribution

Rule: Uniqueness

Common mistakes

Moment generating and characteristic functions

Summary card

Free further reading