5.9.2 Fisher information and standard errors

In this lesson

How sharp the peak of a log-likelihood is, and why that says how precise $\hat{\theta}$ is
The observed information $J = -\ell''(\hat{\theta})$ and the standard error $1/\sqrt{J}$ , with its units
Fisher information: $n$ times the information in one observation, so standard errors shrink like $\frac{1}{\sqrt{n}}$
Several parameters: the Hessian, its inverse, and why estimates that trade off are less certain
A check by simulation, and where the approximation is poor
Confidence intervals from the standard error, where they break, and the likelihood-ratio interval

You need the log-likelihood and maximum likelihood estimates from lesson 5.9.1 and the second derivative from lesson 2.4.2. You also need the Hessian and the quadratic copy from lesson 4.3.1, and the matrix inverse from lesson 3.2.3. Lesson 5.8.2 read a 95% interval as one that covers the true value in 95% of samples.

A sharp peak or a flat one

Lesson 5.9.1 found $\hat{p} = 0.7$ from 7 heads in 10 tosses. Seventy heads in 100 tosses give the same estimate. But 100 tosses pin $p$ down far better, and the log-likelihood shows how: its peak is sharper.

Here is how far $\ell(p) = k\ln p + (n - k)\ln(1 - p)$ falls below its peak at a few values of $p$ .

$p$	0.5	0.6	0.65	0.75	0.8
drop, 7 heads in 10	0.8228	0.216	0.0563	0.064	0.2817
drop, 70 heads in 100	8.2283	2.1601	0.563	0.6401	2.8168

Ten times the data, ten times the drop, at every $p$ . A drop of 8.2283 means the data are $e^{8.2283} \approx 3{,}745$ times less likely under a fair coin than under $p = 0.7$ . With 10 tosses the factor is only 2.28.

The log-likelihood below its peak for 10 and 100 tosses, with the parabola its curvature gives (dashed). The same peak, ten times the curvature. The dotted line 1.92 below the peak marks the interval of the last section.

The curvature at the peak

Near its peak, $\ell$ is close to its quadratic copy (lesson 2.7.1). At $\hat{\theta}$ the slope is zero, so only the bend term is left:

\ell(\theta) \approx \ell(\hat{\theta}) + \tfrac{1}{2}\,\ell''(\hat{\theta})\,(\theta - \hat{\theta})^2

$\ell''(\hat{\theta})$ is negative at a peak (lesson 2.4.2). The bigger its size, the faster $\ell$ falls away, and the fewer values of $\theta$ explain the data almost as well as $\hat{\theta}$ .

Observed information and the standard error

The size of the bend at the peak is called the observed information, $J = -\ell''(\hat{\theta})$ . Its inverse square root is the standard error of $\hat{\theta}$ : the typical distance between the estimate and the true value.

Rule: Standard error from the curvature

\begin{gathered} J = -\ell''(\hat{\theta}) \\[4pt] \operatorname{SE}(\hat{\theta}) \approx \frac{1}{\sqrt{J}} \end{gathered}

$J$ is in units of 1 over $\theta$ 's units squared, so the standard error is in the units of $\theta$ . At one standard error from $\hat{\theta}$ the quadratic copy has dropped by $\frac{1}{2}$ .

For the coin, lesson 5.9.1 found $\ell''(p) = -\frac{k}{p^2} - \frac{n - k}{(1 - p)^2}$ . At $\hat{p} = \frac{k}{n}$ this simplifies to

\begin{gathered} J = \frac{n}{\hat{p}(1 - \hat{p})} \\[6pt] \operatorname{SE}(\hat{p}) = \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} \end{gathered}

data	$J$	standard error
7 heads in 10	47.62	0.1449
70 heads in 100	476.19	0.0458

This is the standard error that case study 4.7.5 attached to a share of hits; maximum likelihood gives the same formula. The quadratic copy, $\frac{1}{2}J(p - 0.7)^2$ , gives drops of 0.2381 at 0.6 and at 0.8, close to the true 0.216 and 0.2817.

J

import math def ell(p, k, n): return k * math.log(p) + (n - k) * math.log(1 - p) for p in [0.5, 0.6, 0.65, 0.75, 0.8]: drops = [round(ell(0.7, k, n) - ell(p, k, n), 4) for k, n in [(7, 10), (70, 100)]] print(p, drops, round(0.5 * (10 / 0.21) * (p - 0.7) ** 2, 4)) print(round(math.exp(ell(0.7, 70, 100) - ell(0.5, 70, 100))), round(math.exp(ell(0.7, 7, 10) - ell(0.5, 7, 10)), 2)) d = 1e-4 for k, n in [(7, 10), (70, 100)]: J = n / (0.7 * 0.3) J_num = -(ell(0.7 + d, k, n) - 2 * ell(0.7, k, n) + ell(0.7 - d, k, n)) / d**2 print(n, round(J, 2), round(J_num, 2), round(1 / math.sqrt(J), 4))

⌘/Ctrl + Enter runs the cell

Fisher information

Before any data arrive, the curvature is random: it depends on the sample. Its expected value is the Fisher information. For $n$ independent observations the log-likelihood is a sum of $n$ terms, so the information is $n$ times the information in one observation.

Rule: Fisher information

\begin{gathered} \mathcal{I}(\theta) = -\mathbb{E}\big[\ell''(\theta)\big] = n\,\mathcal{I}_1(\theta) \\[4pt] \operatorname{SE}(\hat{\theta}) \approx \frac{1}{\sqrt{n\,\mathcal{I}_1(\hat{\theta})}} \end{gathered}

$\mathcal{I}_1$ is the information in one observation. Information adds up over independent data, so the standard error shrinks like $\frac{1}{\sqrt{n}}$ : four times the data halve it. For the three models below, $J$ and $\mathcal{I}(\hat{\theta})$ are equal.

model	$\mathcal{I}_1$	standard error
coin, $p$	$\frac{1}{p(1 - p)}$	$\sqrt{\hat{p}(1 - \hat{p})/n}$
exponential, $\lambda$	$\frac{1}{\lambda^2}$	$\hat{\lambda}/\sqrt{n}$
normal, $\mu$	$\frac{1}{\sigma^2}$	$\hat{\sigma}/\sqrt{n}$
normal, $\sigma$	$\frac{2}{\sigma^2}$	$\hat{\sigma}/\sqrt{2n}$

For one toss, $\ell''(p) = -\frac{X}{p^2} - \frac{1 - X}{(1 - p)^2}$ with $X$ the 0 or 1 it shows. Its mean, with $\mathbb{E}[X] = p$ , is $-\frac{1}{p} - \frac{1}{1 - p} = -\frac{1}{p(1 - p)}$ . For one exponential wait, $\ell''(\lambda) = -\frac{1}{\lambda^2}$ whatever the data.

The waits of lesson 5.9.1, five of them with $\hat{\lambda} = 0.5$ per minute, have a standard error of $\frac{0.5}{\sqrt{5}} \approx 0.2236$ per minute. The normal's $\hat{\sigma}/\sqrt{n}$ is the $\frac{\sigma}{\sqrt{n}}$ of a sample mean that case study 5.3.5 used.

\ell''(0.7)

import math import numpy as np p = 0.7 X = (np.random.default_rng(5921).random(100_000) < p).astype(float) second = -X / p**2 - (1 - X) / (1 - p) ** 2 print(round(second.mean(), 4), round(-1 / (p * (1 - p)), 4)) waits = np.array([2, 0.5, 1.5, 4, 2]) lam = 1 / waits.mean() se = lam / math.sqrt(len(waits)) print(lam, round(se, 4), round(lam - 1.96 * se, 3), round(lam + 1.96 * se, 3))

⌘/Ctrl + Enter runs the cell

Several parameters: the Hessian

With several parameters the curvature is the Hessian $H$ of $\ell$ at $\hat{\boldsymbol{\theta}}$ (lesson 4.3.1), with $\boldsymbol{\theta}$ the vector of parameters. The observed information matrix is $J = -H$ , the Hessian of the negative log-likelihood. Its inverse approximates the covariance matrix of the estimates.

Rule: Standard errors from the Hessian

\begin{gathered} J = -H(\hat{\boldsymbol{\theta}}) \\[4pt] \operatorname{Cov}(\hat{\boldsymbol{\theta}}) \approx J^{-1} \\[4pt] \operatorname{SE}(\hat{\theta}_i) \approx \sqrt{(J^{-1})_{ii}} \end{gathered}

Take the diagonal of the inverse, not 1 over the diagonal. The two agree only when $J$ is diagonal; otherwise the inverse gives larger standard errors.

For the normal fit of lesson 5.9.1 (data 2, 4, 5, 6, 8), $J = \begin{pmatrix} 1.25 & 0 \\ 0 & 2.5 \end{pmatrix}$ . Its inverse is $\begin{pmatrix} 0.8 & 0 \\ 0 & 0.4 \end{pmatrix}$ , so $\operatorname{SE}(\hat{\mu}) = \sqrt{0.8} \approx 0.8944$ and $\operatorname{SE}(\hat{\sigma}) = \sqrt{0.4} \approx 0.6325$ . These are $\frac{2}{\sqrt{5}}$ and $\frac{2}{\sqrt{10}}$ , the table's formulas.

When estimates trade off

Lessons 3.5.3 and 4.4.1 fitted the line $y = 1 + 1.2x$ to five points by least squares. There $M$ is the matrix whose columns are the ones and the $x$ values.

Suppose each $y$ has a normal error with standard deviation $\sigma$ . Then $\ell$ is $-\frac{1}{2\sigma^2}$ times the sum of squared misses, plus terms without $a$ and $b$ , so its peak is the least-squares line.

Take $\sigma = 0.4$ , the fit's own estimate $\sqrt{0.8/5}$ . The Hessian of $-\ell$ in the intercept $a$ and slope $b$ is $\frac{1}{\sigma^2}M^\top M$ :

\begin{gathered} J = \begin{pmatrix} 31.25 & 62.5 \\ 62.5 & 187.5 \end{pmatrix} \\[6pt] J^{-1} = \begin{pmatrix} 0.096 & -0.032 \\ -0.032 & 0.016 \end{pmatrix} \end{gathered}

The standard errors are $\sqrt{0.096} \approx 0.3098$ for $a$ and $\sqrt{0.016} \approx 0.1265$ for $b$ . One over the square root of the diagonal would give 0.1789 and 0.073, far too small.

The reason is the off-diagonal entry. A steeper slope with a lower intercept fits almost as well, so the two estimates move together, with correlation $-0.8165$ . Lesson 5.10.3 meets this matrix again as the covariance of regression estimates.

J

import numpy as np J = np.array([[1.25, 0.0], [0.0, 2.5]]) print(np.linalg.inv(J), np.round(np.sqrt(np.diag(np.linalg.inv(J))), 4)) x = np.array([0, 1, 2, 3, 4.0]) M = np.column_stack([np.ones(5), x]) J = M.T @ M / 0.4**2 C = np.linalg.inv(J) print(J, np.round(C, 4)) print(np.round(np.sqrt(np.diag(C)), 4), np.round(1 / np.sqrt(np.diag(J)), 4), round(C[0, 1] / np.sqrt(C[0, 0] * C[1, 1]), 4))

⌘/Ctrl + Enter runs the cell

Checked by simulation

The standard error claims to be the spread of $\hat{\theta}$ over many samples. A simulation can test that. Draw many samples from a model with a known $\lambda = 2$ , compute $\hat{\lambda} = \frac{1}{\bar{x}}$ for each, and compare their standard deviation with $\frac{\lambda}{\sqrt{n}}$ .

$n$	10	30	100	1,000
mean of $\hat{\lambda}$	2.2178	2.0734	2.0195	2.0025
spread of $\hat{\lambda}$	0.7956	0.3956	0.2034	0.0627
$\lambda/\sqrt{n}$	0.6325	0.3651	0.2	0.0632

With 10 waits, the true spread is a quarter larger than the formula, and $\hat{\lambda}$ is biased upwards (lesson 5.8.1). From about 100 on, both match well. The numbers are from one seeded run of 10,000 samples for each $n$ .

Rule: Large samples

\hat{\theta} \approx \mathcal{N}\!\left(\theta,\ \frac{1}{\mathcal{I}(\theta)}\right)

Stated, not proved: for large $n$ , a maximum likelihood estimate is close to normal, centred on the true $\theta$ , with variance $\frac{1}{\mathcal{I}(\theta)}$ . Like the central limit theorem of lesson 5.7.3, it says nothing exact about a small sample.

n

import math import numpy as np lam = 2 rng = np.random.default_rng(592) for n in [10, 30, 100, 1000]: X = rng.exponential(1 / lam, size=(10_000, n)) est = 1 / X.mean(axis=1) covered = np.abs(est - lam) < 1.96 * est / math.sqrt(n) print(n, round(est.mean(), 4), round(est.std(), 4), round(lam / math.sqrt(n), 4), round(covered.mean(), 4))

⌘/Ctrl + Enter runs the cell

Confidence intervals

If $\hat{\theta}$ is close to normal with standard deviation $\operatorname{SE}$ , then $\hat{\theta} \pm 1.96\operatorname{SE}$ covers the true value about 95% of the time, as in lesson 5.8.2.

Rule: Interval from the standard error

\hat{\theta} \pm 1.96\,\operatorname{SE}(\hat{\theta})

An approximate 95% interval, called a Wald interval. For 90% or 99%, use 1.645 or 2.576 in place of 1.96 (lesson 5.4.3).

7 heads in 10: $0.7 \pm 1.96 \times 0.1449$ , from 0.416 to 0.984.
70 heads in 100: from 0.6102 to 0.7898. The width is $\frac{1}{\sqrt{10}} \approx 0.32$ times the first, because the information grew tenfold.
The waits: $0.5 \pm 1.96 \times 0.2236$ , from 0.062 to 0.938 per minute.

The simulation above found the exponential intervals covering 2 in 95.01% to 95.29% of samples, close to 95% at every $n$ . For the coin with $p = 0.7$ , a seeded run gives 84.14% with 10 tosses and 95.04% with 100.

Where these intervals break

Near the edge of the range the Wald interval misbehaves. With 9 heads in 10 it runs from 0.714 to 1.086, past 1.

With 10 heads in 10, $\hat{p}(1 - \hat{p}) = 0$ , so the standard error is 0. The interval is the single point 1: it claims certainty from 10 tosses.

The cause is the quadratic copy. Near an edge the log-likelihood is far from a parabola, so its curvature at the peak says little about its shape further out.

The likelihood-ratio interval

A sounder interval keeps every $\theta$ whose log-likelihood is within 1.92 of the peak. The number is $\frac{1}{2} \times 3.841$ , where $3.841 \approx 1.96^2$ is the 0.95-quantile of $\chi^2_1$ (lesson 5.4.4). For an exact parabola this is the Wald interval again.

Rule: Likelihood-ratio interval

\Big\{\theta : \ell(\theta) \ge \ell(\hat{\theta}) - 1.92\Big\}

An approximate 95% interval read off the log-likelihood itself, not its parabola. It need not be symmetric about $\hat{\theta}$ , and it stays inside the range of $\theta$ .

data	Wald	likelihood ratio
7 heads in 10	0.416 to 0.984	0.3935 to 0.9154
70 heads in 100	0.6102 to 0.7898	0.6059 to 0.784
9 heads in 10	0.714 to 1.086	0.6284 to 0.994
10 heads in 10	1 to 1	0.8252 to 1

With 100 tosses the two nearly agree. With 10 they differ, and the likelihood-ratio interval leans towards the middle, where the log-likelihood falls more slowly.

For 10 tosses with $p = 0.7$ it covers the truth with probability 0.9612, against 0.8401 for the Wald interval. Case study 5.9.4 uses it for the degrees of freedom of a t.

k

import math import numpy as np from scipy.optimize import brentq from scipy.stats import binom, chi2 def ell(p, k, n): return (k * math.log(p) if k else 0) + ((n - k) * math.log(1 - p) if n - k else 0) def wald(k, n): p = k / n se = math.sqrt(p * (1 - p) / n) return round(p - 1.96 * se, 4), round(p + 1.96 * se, 4) def lr(k, n): p, c = k / n, chi2.ppf(0.95, 1) / 2 gap = lambda q: ell(q, k, n) - ell(p, k, n) + c lo = 0.0 if k == 0 else brentq(gap, 1e-12, p) hi = 1.0 if k == n else brentq(gap, p, 1 - 1e-12) return round(lo, 4), round(hi, 4) print(round(chi2.ppf(0.95, 1), 3), round(chi2.ppf(0.95, 1) / 2, 2)) for k, n in [(7, 10), (70, 100), (9, 10), (10, 10)]: print(k, n, wald(k, n), lr(k, n)) rng = np.random.default_rng(593) for n in [10, 100]: p_hat = rng.binomial(n, 0.7, size=100_000) / n se = np.sqrt(p_hat * (1 - p_hat) / n) print(n, round(float(np.mean(np.abs(p_hat - 0.7) < 1.96 * se)), 4)) def covers(interval): return interval[0] <= 0.7 <= interval[1] print(round(sum(binom.pmf(k, 10, 0.7) for k in range(11) if covers(lr(k, 10))), 4), round(sum(binom.pmf(k, 10, 0.7) for k in range(11) if covers(wald(k, 10))), 4))

⌘/Ctrl + Enter runs the cell

Common mistakes

The standard error is $\frac{1}{\sqrt{J}}$ , not $\frac{1}{J}$ and not $\sqrt{J}$ . For 7 heads in 10, $J = 47.62$ and the standard error is 0.1449.
Take the curvature at the peak, of $\ell$ , not of $L$ . Elsewhere $\ell''$ is a different number.
More data, smaller error, but slowly. The information grows like $n$ , the standard error shrinks like $\frac{1}{\sqrt{n}}$ : four times the data halve it.
With several parameters, invert first. The standard errors are the square roots of the diagonal of $J^{-1}$ . For the line they are 0.3098 and 0.1265, not 0.1789 and 0.073.
The formula is for large samples. With 10 waits the true spread of $\hat{\lambda}$ is 0.7956, against 0.6325 from the formula.
Wald intervals break at the edges. 10 heads in 10 give a standard error of 0. Use the likelihood-ratio interval there.

Fisher information and standard errors

0 of 12 correct

1.
A coin shows 7 heads in 10 tosses. Find the observed information $J = \frac{n}{\hat{p}(1 - \hat{p})}$ (a fraction)
2.
For the same data, find the standard error of $\hat{p}$ (type sqrt)
3.
A coin shows 64 heads in 100 tosses. Find the standard error of $\hat{p}$
4.
At the peak of a log-likelihood, $\ell''(\hat{\theta}) = -400$ . Find the standard error of $\hat{\theta}$
5.
25 exponential waits have mean 4 minutes. Find the standard error of $\hat{\lambda}$ , per minute
6.
A normal fit to 100 points gives $\hat{\sigma} = 2$ . Find the standard error of $\hat{\sigma}$ (type sqrt)
7.
An estimate from 400 observations has standard error 0.1. Estimate its standard error from 1,600 observations
8.
A log-likelihood has $J = 50$ at its peak. By the quadratic copy, how far does $\ell$ fall at 0.2 from the peak?
9.
A two-parameter fit has $J = \begin{pmatrix} 4 & 2 \\ 2 & 3 \end{pmatrix}$ . Find the standard error of the first parameter (type sqrt)
10.
Find the upper end of the Wald interval $\hat{p} + 1.96\operatorname{SE}$ for 64 heads in 100 tosses
11.
A coin shows 10 heads in 10 tosses. What is the width of the Wald interval for $p$ ?
12.
A 95% likelihood-ratio interval keeps every $\theta$ within how much of the peak of $\ell$ ? Use $1.96^2$ for the $\chi^2_1$ quantile

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\ell(\theta) \approx \ell(\hat{\theta}) - \frac{1}{2}J(\theta - \hat{\theta})^2

Fisher information and standard errors

A sharp peak or a flat one

The curvature at the peak

Observed information and the standard error

Rule: Standard error from the curvature

Fisher information

Rule: Fisher information

Several parameters: the Hessian

Rule: Standard errors from the Hessian

When estimates trade off

Checked by simulation

Rule: Large samples

Confidence intervals

Rule: Interval from the standard error

Where these intervals break

The likelihood-ratio interval

Rule: Likelihood-ratio interval

Common mistakes

Fisher information and standard errors

Summary card

Free further reading