5.1.4 Bayes' rule | SigmaField

Reversing a condition

Lesson 5.1.3 had a bag with a fair coin and a biased coin that lands heads with probability 0.8. One coin is picked at random and tossed, and it shows heads. Which coin was it?

You know the chance of heads for each coin. You want the reverse: the chance of each coin, given heads. The multiplication rule (lesson 5.1.3) writes $P(\text{biased} \cap H)$ in two ways:

\begin{gathered} P(\text{biased} \mid H)\,P(H) \\ = P(H \mid \text{biased})\,P(\text{biased}) \end{gathered}

Divide by $P(H) = 0.65$ , which lesson 5.1.3 found by the law of total probability. Then $P(\text{biased} \mid H) = \frac{0.8 \times 0.5}{0.65} = \frac{8}{13} \approx 0.6154$ . Heads moved the biased coin from 0.5 to 0.6154.

The table of counts from lesson 5.1.3 agrees. Of the 650 heads in 1,000 repeats, 400 came from the biased coin, and $\frac{400}{650} = \frac{8}{13}$ .

Rule: Bayes' rule

\begin{gathered} P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)} \\[4pt] P(B) = P(B \mid A)\,P(A) \\ +\ P(B \mid A^c)\,P(A^c) \end{gathered}

It turns $P(B \mid A)$ into $P(A \mid B)$ . The bottom is the law of total probability over " $A$ " and "not $A$ ". It needs $P(B) > 0$ .

Tails works the same way: $P(\text{biased} \mid T) = \frac{0.2 \times 0.5}{0.35} = \frac{2}{7} \approx 0.2857$ . Evidence can push a probability either way.

Prior, likelihood and posterior

Bayes' rule has names for its parts.

The prior $P(A)$ : how likely $A$ is before the evidence. Here, 0.5 for the biased coin.
The likelihood $P(B \mid A)$ : how likely the evidence is if $A$ is true. Heads has likelihood 0.8 for the biased coin and 0.5 for the fair one.
The posterior $P(A \mid B)$ : how likely $A$ is after the evidence. Here, $\frac{8}{13}$ .

In words: multiply each prior by its likelihood, then divide by the total of those products. The total is $P(B)$ , which makes the posteriors add up to 1.

The same steps work with more than two cases. A box holds three coins: fair, biased (heads with 0.8) and two-headed. One is picked at random and shows heads.

coin	prior	likelihood of heads	prior × likelihood	posterior
fair	$\frac{1}{3}$	0.5	0.1667	0.2174
biased	$\frac{1}{3}$	0.8	0.2667	0.3478
two-headed	$\frac{1}{3}$	1	0.3333	0.4348
total	1		0.7667	1

The total, 0.7667, is $P(H)$ . Each posterior is its row's product divided by it: $\frac{5}{23}$ , $\frac{8}{23}$ and $\frac{10}{23}$ . With equal priors, the posteriors are in the ratio of the likelihoods.

Try it bayes takes a list of priors and a list of likelihoods, one per case, and returns the posteriors and their total. The first two lines are the two coins after heads and after tails; the rest is the table above.

from fractions import Fraction as F def bayes(priors, likelihoods): products = [p * l for p, l in zip(priors, likelihoods)] total = sum(products) return [x / total for x in products], total post, total = bayes([F(1, 2), F(1, 2)], [F(4, 5), F(1, 2)]) print(*post, total) post, total = bayes([F(1, 2), F(1, 2)], [F(1, 5), F(1, 2)]) print(*post, total) likelihoods = [F(1, 2), F(4, 5), F(1)] post, total = bayes([F(1, 3)] * 3, likelihoods) print([round(float(F(1, 3) * l), 4) for l in likelihoods], round(float(total), 4)) print(*post, [round(float(x), 4) for x in post])

⌘/Ctrl + Enter runs the cell

The base-rate trap

A condition affects 1% of people: that is its base rate, the prior for someone picked at random. A test is positive for 90% of the people who have it, its hit rate. It is also positive for 5% of the people who do not, its false-alarm rate.

Someone tests positive. Many people guess they have the condition with probability 90%. Bayes' rule says:

\begin{aligned} P(\text{has it} \mid +) &= \frac{0.9 \times 0.01}{0.9 \times 0.01 + 0.05 \times 0.99} \\ &= \frac{0.009}{0.0585} \approx 0.1538 \end{aligned}

About 15%: most positives are false alarms. A table of counts for 10,000 people shows why.

	positive	negative	total
has it	90	10	100
does not	495	9,405	9,900
total	585	9,415	10,000

Only 100 people have the condition, and 90 of them test positive. But 5% of the 9,900 others is 495 false alarms. So 90 of the 585 positives have it: $\frac{90}{585} = \frac{2}{13} \approx 0.1538$ .

The guess of 90% swaps the condition: it is $P(+ \mid \text{has it})$ , not $P(\text{has it} \mid +)$ . Ignoring the base rate in this way is the base-rate trap. A negative, on the other hand, is very reassuring: $\frac{10}{9{,}415} \approx 0.00106$ .

The same test means different things in different groups. At a 1% base rate, a positive from the 90% and 5% test leaves the condition at 0.1538; a sharper test reaches 0.5.

The rarer the condition, the more the false alarms dominate. At a 0.1% base rate the test gives only 0.0177; at 10% it gives 0.6667. Case study 5.1.6 meets the same effect with an early-warning signal for loan defaults.

Try it posterior is Bayes' rule with a hit rate and a false-alarm rate. The first lines give the positive and negative cases and the table of counts. Then a seeded simulation of a million people: each has the condition with probability 0.01, and tests positive with probability 0.9 or 0.05.

import numpy as np def posterior(prior, hit, false_alarm): return hit * prior / (hit * prior + false_alarm * (1 - prior)) print(round(0.9 * 0.01, 4), round(0.05 * 0.99, 4), round(0.9 * 0.01 + 0.05 * 0.99, 4)) print(round(posterior(0.01, 0.9, 0.05), 4), round(0.1 * 0.01 / (0.1 * 0.01 + 0.95 * 0.99), 5)) print(10_000 * 0.01 * 0.9, 9_900 * 0.05, round(90 / 585, 4)) print([round(posterior(b, 0.9, 0.05), 4) for b in [0.001, 0.01, 0.1]], posterior(0.01, 0.99, 0.01)) rng = np.random.default_rng(7) has_it = rng.random(1_000_000) < 0.01 u = rng.random(1_000_000) positive = np.where(has_it, u < 0.9, u < 0.05) print(has_it.sum(), positive.sum(), round(has_it[positive].mean(), 4))

⌘/Ctrl + Enter runs the cell

In the seeded run, 9,955 people had the condition and 58,466 tested positive. Among the positives, a share 0.1536 had it, against the exact 0.1538.

Updating twice

After a positive, the doctor orders a second test of the same kind. Suppose its result is independent of the first, once you know whether the person has the condition. Then the first posterior, 0.1538, is the prior for the second test:

\frac{0.9 \times 0.1538}{0.9 \times 0.1538 + 0.05 \times 0.8462} \approx 0.766

Two positives give 0.766, and a third would give 0.9833. Doing both tests in one step, with likelihoods $0.9^2$ and $0.05^2$ , gives the same 0.766.

The independence matters. If the second test shares the first one's errors, say because a cause of false alarms affects both, it adds much less than this.

The odds form

The odds of $A$ are $\frac{P(A)}{P(A^c)}$ : a probability of 0.01 is odds of $\frac{1}{99}$ , "1 to 99". Odds $o$ turn back into a probability $\frac{o}{1 + o}$ . Bayes' rule is shortest in odds.

Rule: Bayes' rule in odds

\begin{gathered} \text{posterior odds} = \text{LR} \times \text{prior odds} \\[4pt] \text{LR} = \frac{P(B \mid A)}{P(B \mid A^c)} \end{gathered}

The likelihood ratio LR says how many times more likely the evidence is if $A$ is true than if it is not. For a positive test, it is the hit rate over the false-alarm rate.

For the test, $\text{LR} = \frac{0.9}{0.05} = 18$ . Prior odds $\frac{1}{99}$ become $\frac{18}{99} = \frac{2}{11}$ , a probability of $\frac{2}{13} \approx 0.1538$ . A second positive multiplies by 18 again: $\frac{36}{11}$ , or $\frac{36}{47} \approx 0.766$ .

When evidence cannot move a probability

LR = 1: a test that is positive as often with the condition as without it tells you nothing. The posterior equals the prior.
No false alarms: if $P(B \mid A^c) = 0$ , a positive rules out "not $A$ ", and the posterior is 1.
A prior of 0 or 1 never moves: $0 \times \text{LR} = 0$ . Give a prior of 0 only to what is truly impossible.

Try it Two positives, by updating twice and in one step, then the odds form. The last line runs the three cases above: equal rates, no false alarms, and a prior of 0.

def posterior(prior, hit, false_alarm): return hit * prior / (hit * prior + false_alarm * (1 - prior)) once = posterior(0.01, 0.9, 0.05) twice = posterior(once, 0.9, 0.05) print(round(once, 4), round(1 - once, 4), round(twice, 4), round(posterior(twice, 0.9, 0.05), 4)) print(round(posterior(0.01, 0.9**2, 0.05**2), 4)) odds = (0.01 / 0.99) * (0.9 / 0.05) print(round(odds, 4), round(odds / (1 + odds), 4), round(odds * 18 / (1 + odds * 18), 4)) print(posterior(0.3, 0.6, 0.6), posterior(0.01, 0.9, 0.0), posterior(0.0, 0.9, 0.05))

⌘/Ctrl + Enter runs the cell

Common mistakes

Swapping the condition. $P(+ \mid \text{has it}) = 0.9$ is the hit rate; $P(\text{has it} \mid +) \approx 0.15$ is what a patient wants.
Ignoring the base rate. The same positive means 0.0177 at a 0.1% base rate and 0.6667 at 10%.
Leaving out the false alarms. The bottom of Bayes' rule adds every way the evidence can happen: $0.009 + 0.0495$ , not just $0.009$ .
Hit and false-alarm rates need not add up to 1. They describe different groups: 0.9 and 0.05 here.
Updating twice on shared errors. Two tests multiply their likelihood ratios only if their results are independent given the truth.
A prior of 0 cannot be updated. No evidence moves it, so keep small but non-zero priors for rare things.

Bayes' rule

0 of 12 correct

1.
$P(B \mid A) = 0.6$ , $P(A) = 0.2$ and $P(B) = 0.3$ . Find $P(A \mid B)$
2.
A bag holds a fair coin and a two-headed coin. One is picked at random and shows heads. Find the probability that it is the two-headed coin
3.
Machine A makes 60% of the parts, 2% of them faulty; machine B makes 40%, 5% faulty. A part is faulty. Find the probability that machine B made it
4.
A condition has a base rate of 2%. A test has a 90% hit rate and a 10% false-alarm rate. Find $P(\text{has it} \mid +)$
5.
Of 1,000 people, 50 have a condition. A test is positive for 40 of them and for 95 of the other 950. What share of the positives have the condition?
6.
Three coins, fair, biased (heads with 0.8) and two-headed, are equally likely to be picked. The coin shows heads. Find the probability that it is the fair coin
7.
40% of emails are spam. A word appears in 30% of spam and 2% of other emails. Find $P(\text{spam} \mid \text{word})$
8.
After one positive test the probability is 0.25. A second, independent test has a 90% hit rate and a 10% false-alarm rate. Find the probability after a second positive
9.
A probability of 0.2 is what odds?
10.
The prior odds are 1 to 4 and the likelihood ratio is 3. Find the posterior probability
11.
A test is positive for 30% of those with a condition and 30% of those without. The prior is 0.1. Find the posterior after a positive
12.
A test never gives false alarms. The prior is 0.01 and the test is positive. Find the posterior

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}

Bayes' rule

Reversing a condition

Rule: Bayes' rule

Prior, likelihood and posterior

The base-rate trap

Updating twice

The odds form

Rule: Bayes' rule in odds

When evidence cannot move a probability

Common mistakes

Bayes' rule

Summary card

Free further reading