5.1.3 Conditional probability and independence

Knowing something shrinks the sample space

Roll two fair dice, red and blue. The red die shows 6 with probability $\frac{1}{6}$ . Now someone tells you that the sum is at least 9: what is the chance of a red 6 now?

Only the 10 outcomes with a sum of 9 or more are still possible, and they are still equally likely. Four of them have a red 6: $(6, 3)$ , $(6, 4)$ , $(6, 5)$ and $(6, 6)$ . So the chance is now $\frac{4}{10} = 0.4$ .

Knowing $B$ , a sum of at least 9, leaves only its 10 shaded cells. The red 6 takes 6 of the 36 cells of the grid, but 4 of the 10 cells of $B$ .

News can also lower a chance, or leave it alone. If the sum is at most 6, a red 6 is impossible, since $6 + 1 = 7$ . If the sum is exactly 7, one of its 6 outcomes, $(6, 1)$ , has a red 6: the chance stays $\frac{1}{6}$ .

The definition

Shrinking to $B$ keeps only the part of $A$ inside $B$ . Dividing by the size of $B$ makes the probabilities on $B$ add up to 1 again.

Rule: Conditional probability

P(A \mid B) = \frac{P(A \cap B)}{P(B)}

Read "the probability of $A$ given $B$ ". It is defined only when $P(B) > 0$ : you cannot condition on something impossible. With equally likely outcomes it is $\frac{|A \cap B|}{|B|}$ .

For the dice, $P(A \cap B) = \frac{4}{36}$ and $P(B) = \frac{10}{36}$ , so $P(A \mid B) = \frac{4}{10}$ . Conditional probabilities follow the rules of lesson 5.1.1, on the smaller sample space: $P(A^c \mid B) = 1 - 0.4 = 0.6$ .

The order matters. $P(B \mid A)$ asks for a sum of at least 9 given a red 6. Then the blue die needs 3 or more, so $P(B \mid A) = \frac{4}{6} = \frac{2}{3}$ , not 0.4.

The two share the top, $P(A \cap B)$ , but divide by different events. Lesson 5.1.4 shows how costly a swap can be.

P(a \mid b)

from fractions import Fraction from itertools import product dice = list(product(range(1, 7), repeat=2)) def P(event): return Fraction(sum(event(w) for w in dice), len(dice)) def cond(a, b): return Fraction(sum(a(w) and b(w) for w in dice), sum(b(w) for w in dice)) red6 = lambda w: w[0] == 6 big = lambda w: w[0] + w[1] >= 9 print(P(red6), P(big), P(lambda w: red6(w) and big(w))) print(cond(red6, big), cond(big, red6)) print(cond(red6, lambda w: w[0] + w[1] <= 6), cond(red6, lambda w: w[0] + w[1] == 7))

⌘/Ctrl + Enter runs the cell

In a simulation: keep only the runs where $B$ happened

Conditioning has a direct meaning in a simulation. Throw away every run in which $B$ did not happen, then count $A$ among the rest.

With the seed 4, a million rolls of two dice gave 278,491 with a sum of at least 9. Of those, a share 0.3991 had a red 6, close to 0.4. Among the rolls with a sum of 7, the share was 0.1674, close to $\frac{1}{6}$ .

Try it rolls[keep, 0] takes the red die of the rows where keep is True, so .mean() of the test is the conditional share.

import numpy as np rng = np.random.default_rng(4) rolls = rng.integers(1, 7, size=(1_000_000, 2)) s = rolls.sum(axis=1) keep = s >= 9 print(keep.sum(), round((rolls[keep, 0] == 6).mean(), 4)) print(round((rolls[s == 7, 0] == 6).mean(), 4))

⌘/Ctrl + Enter runs the cell

The multiplication rule

Multiply the definition through by $P(B)$ and it gives the chance that both events happen.

Rule: Multiplication rule

\begin{gathered} P(A \cap B) = P(A)\,P(B \mid A) \\ = P(B)\,P(A \mid B) \end{gathered}

The chance of $A$ , times the chance of $B$ once $A$ has happened. For three events, $P(A \cap B \cap C) = P(A)\,P(B \mid A)\,P(C \mid A \cap B)$ , and so on.

It suits draws without replacement, where each conditional chance can be read off. An urn holds 3 red and 2 blue balls, and two are drawn.

The first is red with probability $\frac{3}{5}$ . Then 2 of the 4 balls left are red, so $P(\text{both red}) = \frac{3}{5} \times \frac{2}{4} = \frac{3}{10}$ , as lesson 5.1.2 found by counting.

The same works for cards. Two aces in two cards: $\frac{4}{52} \times \frac{3}{51} = \frac{1}{221}$ . Three hearts in a row: $\frac{13}{52} \times \frac{12}{51} \times \frac{11}{50} = \frac{11}{850} \approx 0.0129$ .

Trees and the law of total probability

A tree draws an experiment in stages. Each branch carries a conditional probability, given the path so far. Multiply along a path to get the chance of its leaf; add the leaves that make up an event.

Two draws without replacement, with R for red and B for blue. Each leaf's probability is the product along its path. The second ball is red on two paths, $\frac{3}{10} + \frac{3}{10} = \frac{3}{5}$ .

The leaves add up to 1, and so do the branches out of each point: two checks worth making on every tree. The second ball is red with probability $\frac{3}{5}$ , the same as the first. Before you see the first ball, the second has no reason to differ.

That sum over the first draw is a rule of its own. A partition of $\Omega$ is a list of disjoint events $B_1, \ldots, B_n$ that together make up $\Omega$ , like "first red" and "first blue".

Rule: Law of total probability

P(A) = \sum_{i=1}^{n} P(A \mid B_i)\,P(B_i)

Split $\Omega$ into the disjoint pieces $B_1, \ldots, B_n$ , find the chance of $A$ within each, and weight by the size of the piece. Each term is the multiplication rule for $A \cap B_i$ .

A second example. A bag holds a fair coin and a coin that lands heads with probability 0.8. Pick one at random and toss it:

\begin{aligned} P(H) &= P(H \mid \text{fair})\,P(\text{fair}) \\ &\quad + P(H \mid \text{biased})\,P(\text{biased}) \\ &= 0.5 \times 0.5 + 0.8 \times 0.5 = 0.65 \end{aligned}

A table of counts tells the same story. In 1,000 repeats, about 500 pick the fair coin and 250 of those give heads. About 500 pick the biased coin and 400 give heads: 650 heads in all.

coin	picked	heads
fair	500	250
biased	500	400
total	1,000	650

Lesson 5.1.4 asks the reverse question: given heads, which coin was it? Case study 5.1.6 uses the law of total probability to turn one-year default probabilities into two-year ones.

P(R_2 \mid R_1)

from fractions import Fraction as F import numpy as np leaves = {"RR": F(3, 5) * F(2, 4), "RB": F(3, 5) * F(2, 4), "BR": F(2, 5) * F(3, 4), "BB": F(2, 5) * F(1, 4)} print(*leaves.values(), sum(leaves.values()), leaves["RR"] + leaves["BR"]) print(0.5 * 0.5 + 0.8 * 0.5) draws = np.random.default_rng(5).random((200_000, 5)).argsort(axis=1)[:, :2] red = draws < 3 print(round((red[:, 0] & red[:, 1]).mean(), 4), round(red[:, 1].mean(), 4)) print(round(red[red[:, 0], 1].mean(), 4), round(red[~red[:, 0], 1].mean(), 4))

⌘/Ctrl + Enter runs the cell

The seeded run gives 0.2992 for both red and 0.6019 for a red second ball, against 0.3 and 0.6. The conditional shares are 0.5002 and 0.7534.

Independence

Sometimes knowing $B$ changes nothing: $P(A \mid B) = P(A)$ . Then $A$ and $B$ are independent, and the multiplication rule becomes a plain product.

Rule: Independent events

P(A \cap B) = P(A)\,P(B)

$A$ and $B$ are independent when this holds. If $P(B) > 0$ it says the same as $P(A \mid B) = P(A)$ : knowing $B$ does not change the chance of $A$ , and the other way round.

Test it on two dice.

Red 6 and blue 6: $P(\text{both}) = \frac{1}{36} = \frac{1}{6} \times \frac{1}{6}$ . Independent: the dice do not affect each other.
Red 6 and a sum of 7: $P(\text{both}) = \frac{1}{36} = \frac{1}{6} \times \frac{1}{6}$ . Independent too, although the sum uses the red die: whatever red shows, exactly one blue face makes 7.
Red 6 and a sum of at least 9: $P(\text{both}) = \frac{4}{36} = \frac{1}{9}$ , but $\frac{1}{6} \times \frac{10}{36} \approx 0.0463$ . Not independent: a high sum makes a red 6 more likely.

Draws with replacement are independent, because the urn is the same each time. Without replacement they are not: $P(R_2 \mid R_1) = \frac{1}{2}$ , but $P(R_2) = \frac{3}{5}$ .

Several events are independent when every group of them multiplies in this way, as for repeated tosses or rolls. Ten heads in ten fair tosses has probability $\left(\frac{1}{2}\right)^{10} = \frac{1}{1024}$ . At least one six in 4 rolls is $1 - \left(\frac{5}{6}\right)^4 \approx 0.5177$ , by the complement; a million seeded runs below give 0.5171.

Independence is a property of how the experiment runs: separate dice, a fresh shuffle, a ball put back. Do not assume it when one result can affect another.

Independent is not mutually exclusive

Disjoint events sound unrelated, but they are as dependent as events can be. If one happens, the other cannot.

Roll one die, with $A = \{1, 2\}$ and $B = \{5, 6\}$ . They are disjoint, so $P(A \cap B) = 0$ , but $P(A)\,P(B) = \frac{1}{3} \times \frac{1}{3} = \frac{1}{9}$ . Knowing $A$ drops the chance of $B$ from $\frac{1}{3}$ to 0.

Rule: Disjoint events are dependent

\begin{gathered} A \cap B = \varnothing,\ P(A) > 0,\ P(B) > 0 \\ \Rightarrow P(A \cap B) \neq P(A)\,P(B) \end{gathered}

Mutually exclusive events with positive probabilities are never independent. "Mutually exclusive" is about sets that do not overlap; "independent" is about chances that do not change.

P(A \cap B)

from fractions import Fraction from itertools import product import numpy as np dice = list(product(range(1, 7), repeat=2)) def P(event): return Fraction(sum(event(w) for w in dice), len(dice)) def independent(a, b): return P(lambda w: a(w) and b(w)), P(a) * P(b) red6 = lambda w: w[0] == 6 big = lambda w: w[0] + w[1] >= 9 print(*independent(red6, lambda w: w[1] == 6)) print(*independent(red6, lambda w: w[0] + w[1] == 7)) print(*independent(red6, big), round(float(P(red6) * P(big)), 4)) print(Fraction(0), Fraction(2, 6) * Fraction(2, 6)) rolls = np.random.default_rng(6).integers(1, 7, size=(1_000_000, 4)) print(round(1 - (5 / 6)**4, 4), round((rolls == 6).any(axis=1).mean(), 4))

⌘/Ctrl + Enter runs the cell

Common mistakes

$P(A \mid B)$ is not $P(B \mid A)$ . A red 6 given a high sum is 0.4; a high sum given a red 6 is $\frac{2}{3}$ .
Divide by the condition. The event after the bar goes on the bottom: $\frac{P(A \cap B)}{P(B)}$ .
Multiply plain probabilities only for independent events. In general $P(A \cap B) = P(A)\,P(B \mid A)$ . For a red 6 and a sum of at least 9, the product $P(A)\,P(B)$ gives 0.0463 instead of $\frac{1}{9}$ .
Draws without replacement are dependent. After a red ball, the chance of red falls from $\frac{3}{5}$ to $\frac{2}{4}$ .
Disjoint is not independent. Disjoint events with positive probabilities always affect each other.
Check the tree. Branches out of each point add up to 1, and so do the leaves.
No conditioning on the impossible. $P(A \mid B)$ needs $P(B) > 0$ .

Conditional probability and independence

0 of 13 correct

1.
Two fair dice: given that the sum is 4, find the probability that the red die shows 1
2.
$P(A \cap B) = 0.12$ and $P(B) = 0.4$ . Find $P(A \mid B)$
3.
Two fair dice: given that the red die shows 5, find the probability that the sum is at least 10
4.
$P(B) = 0.25$ and $P(A \mid B) = 0.4$ . Find $P(A \cap B)$
5.
Three cards are dealt from a shuffled deck. Find the probability that all three are spades
6.
An urn has 3 red and 2 blue balls. Two are drawn without replacement. Find the probability that the first is blue and the second red
7.
Machine A makes 60% of the parts, 2% of them faulty; machine B makes the rest, 5% faulty. Find the probability that a random part is faulty
8.
A bag holds a fair coin and a coin with heads on both sides. One is picked at random and tossed. Find $P(\text{heads})$
9.
$A$ and $B$ are independent, with $P(A) = 0.3$ and $P(B) = 0.5$ . Find $P(A \cap B)$
10.
With the same $A$ and $B$ , find $P(A \cup B)$
11.
A fair coin is tossed 5 times. Find the probability of at least one head
12.
$P(A) = 0.5$ , $P(B) = 0.4$ and $P(A \cap B) = 0.2$ . Are $A$ and $B$ independent? Type 1 for yes, 0 for no
13.
$A$ and $B$ are disjoint, with $P(A) = 0.2$ and $P(B) = 0.3$ . Find $P(A \mid B)$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

P(A \mid B) = \frac{P(A \cap B)}{P(B)}

Conditional probability and independence

Knowing something shrinks the sample space

The definition

Rule: Conditional probability

In a simulation: keep only the runs where $B$ happened

The multiplication rule

Rule: Multiplication rule

Trees and the law of total probability

Rule: Law of total probability

Independence

Rule: Independent events

Independent is not mutually exclusive

Rule: Disjoint events are dependent

Common mistakes

Conditional probability and independence

Summary card

Free further reading

Knowing something shrinks the sample space

The definition

Rule: Conditional probability

In a simulation: keep only the runs where BBB happened

The multiplication rule

Rule: Multiplication rule

Trees and the law of total probability

Rule: Law of total probability

Independence

Rule: Independent events

Independent is not mutually exclusive

Rule: Disjoint events are dependent

Common mistakes

Summary card

Free further reading

In a simulation: keep only the runs where $B$ happened