5.9.3 Practice set: maximum likelihood

Likelihood and the maximum likelihood estimator

Likelihood and the maximum likelihood estimator (lesson 5.9.1)

0 of 8 correct

1.
A coin shows 3 heads in 5 independent tosses. Find the likelihood at $p = 0.5$ (a fraction)
2.
For the same data, how many times as likely are they under $p = 0.6$ as under $p = 0.5$ ?
3.
For the same data, $\ell(p) = 3\ln p + 2\ln(1 - p)$ . Find the score $\ell'(p)$ at $p = 0.4$ (a fraction)
4.
Waits of 3, 1, 2 and 6 minutes are modelled as exponential with rate $\lambda$ . Find $\hat{\lambda}$ per minute
5.
Four counts 2, 0, 3, 1 are modelled as Poisson with rate $\lambda$ , so $\ell(\lambda) = 6\ln\lambda - 4\lambda$ plus a constant. Find $\hat{\lambda}$
6.
Measurements 3, 5, 7, 9 come from a normal. Find the maximum likelihood estimate $\hat{\sigma}^2$
7.
A normal fit to $n = 4$ points has $\hat{\sigma}^2 = 5$ . Find the Hessian entry $\frac{\partial^2 \ell}{\partial \sigma^2}$ at the peak (a fraction)
8.
A coin shows 6 heads in 6 tosses. Find $\hat{p}$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

Fisher information and standard errors

Fisher information and standard errors (lesson 5.9.2)

0 of 10 correct

1.
A coin shows 3 heads in 5 tosses. Find the observed information $J$ at $\hat{p}$ (a fraction)
2.
A coin shows 20 heads in 100 tosses. Find the standard error of $\hat{p}$
3.
For the same data, find the lower end of the Wald interval $\hat{p} - 1.96\operatorname{SE}$
4.
100 exponential waits have mean 0.5 minutes. Find the standard error of $\hat{\lambda}$ , per minute
5.
A normal fit to 36 points gives $\hat{\sigma} = 3$ . Find the standard error of $\hat{\mu}$
6.
50 Poisson counts have mean 2. For a Poisson, $\ell''(\lambda) = -\frac{\sum k_i}{\lambda^2}$ . Find the standard error of $\hat{\lambda}$
7.
About how many tosses of a nearly fair coin give $\hat{p}$ a standard error of 0.01?
8.
A two-parameter fit has $J = \begin{pmatrix} 9 & 3 \\ 3 & 2 \end{pmatrix}$ . Find the standard error of the second parameter
9.
A 99% likelihood-ratio interval uses $2.576^2$ for the $\chi^2_1$ quantile. Within how much of its peak must $\ell$ stay?
10.
A coin shows no heads in 20 tosses. What standard error does $\sqrt{\hat{p}(1 - \hat{p})/n}$ give?

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

Coding exercise

Exercise

Standard errors from the Hessian

Write standard_errors(nll, theta_hat, d=1e-4), as in lesson 5.9.2. nll takes a NumPy array of parameters and returns the negative log-likelihood; theta_hat is its minimiser, a list or array of one or more numbers.

Build the Hessian of nll at theta_hat by lesson 4.3.1's central differences with bump d.
Raise ValueError if the Hessian is not positive definite (lesson 3.7.1): then theta_hat is not a peak of the likelihood.
Return a NumPy array: the square roots of the diagonal of the inverse Hessian.

The tests use the coin, the waits, the normal and the fitted line of lessons 5.9.1 and 5.9.2. One test has strongly correlated estimates, so 1 over the square root of the diagonal fails. Two more hand you a dip and a saddle.

See the tests

import math

import numpy as np
from scipy.stats import norm, t


def _coin(p):
    return -(7 * math.log(p[0]) + 3 * math.log(1 - p[0]))


def test_coin():
    """7 heads in 10 tosses: the standard error of p at 0.7 is about 0.1449"""
    se = standard_errors(_coin, [0.7])
    expect_close(se, [math.sqrt(0.7 * 0.3 / 10)], what="standard_errors(coin, [0.7])", rel=1e-5)


def test_returns_array():
    """The result is a NumPy array with one entry per parameter"""
    se = standard_errors(_coin, [0.7])
    assert isinstance(se, np.ndarray), f"expected a NumPy array, got {type(se).__name__}"
    expect_equal(se.shape, (1,), what="shape of the result")


def test_waits():
    """Exponential waits 2, 0.5, 1.5, 4, 2: the standard error of the rate 0.5 is about 0.2236"""
    waits = np.array([2, 0.5, 1.5, 4, 2])
    nll = lambda p: -(len(waits) * math.log(p[0]) - p[0] * waits.sum())
    expect_close(standard_errors(nll, [0.5]), [0.5 / math.sqrt(5)], what="standard_errors(waits, [0.5])", rel=1e-5)


def test_normal():
    """A normal fit to 2, 4, 5, 6, 8 at (5, 2): standard errors 2/√5 and 2/√10"""
    x = np.array([2, 4, 5, 6, 8.0])
    nll = lambda p: -norm.logpdf(x, p[0], p[1]).sum()
    expect_close(standard_errors(nll, np.array([5.0, 2.0])), [2 / math.sqrt(5), 2 / math.sqrt(10)], what="standard_errors(normal, [5, 2])", rel=1e-5)


def test_correlated():
    """The line y = 1 + 1.2x with sigma 0.4: standard errors 0.3098 and 0.1265, from the inverse"""
    x = np.array([0, 1, 2, 3, 4.0])
    y = np.array([1, 2, 4, 4, 6.0])
    nll = lambda p: np.sum((y - p[0] - p[1] * x) ** 2) / (2 * 0.4**2)
    expect_close(standard_errors(nll, [1.0, 1.2]), [math.sqrt(0.096), math.sqrt(0.016)], what="standard_errors(line, [1, 1.2])", rel=1e-5)


def test_other_bump():
    """A bump of 0.001 gives the coin's answer too"""
    expect_close(standard_errors(_coin, [0.7], d=1e-3), [math.sqrt(0.021)], what="standard_errors(coin, [0.7], d=1e-3)", rel=1e-5)


def test_dip():
    """At the dip between two peaks, theta = 0 for the data -3 and 3, ValueError"""
    data = np.array([-3.0, 3.0])
    nll = lambda p: -t.logpdf(data, 1, p[0]).sum()
    try:
        standard_errors(nll, [0.0])
    except ValueError:
        return
    raise AssertionError("expected ValueError: at theta = 0 the negative log-likelihood has a maximum, not a minimum")


def test_saddle():
    """At a saddle, x² − y² at (0, 0), ValueError"""
    try:
        standard_errors(lambda p: p[0] ** 2 - p[1] ** 2, [0.0, 0.0])
    except ValueError:
        return
    raise AssertionError("expected ValueError: x² − y² has a saddle at (0, 0), not a minimum")

Likelihood and the maximum likelihood estimator

Fisher information and standard errors

Coding exercise

Free further reading