5.5.1 Joint, marginal and conditional distributions

Two random variables on one experiment

A single experiment can give several numbers at once. Toss three fair coins. Let $X$ count the heads in the first two tosses, and $Y$ the heads in the last two.

The middle coin counts in both, so $X$ and $Y$ are linked.

Each of the 8 equally likely outcomes (lesson 5.1.1) gives one pair $(x, y)$ . Tails, heads, heads gives $X = 1$ and $Y = 2$ ; heads, heads, heads gives $X = 2$ and $Y = 2$ . Counting the outcomes for each pair gives the joint probability mass function $p_{X,Y}(x, y) = P(X = x \text{ and } Y = y)$ , in eighths:

	$y = 0$	$y = 1$	$y = 2$
$x = 0$	1/8	1/8	0
$x = 1$	1/8	2/8	1/8
$x = 2$	0	1/8	1/8

Row $x$ and column $y$ hold $P(X = x, Y = y)$ ; the comma reads "and". The pair $(0, 2)$ cannot happen: no heads in the first two tosses leaves at most one head in the last two.

Rule: Joint pmf

\begin{gathered} p_{X,Y}(x, y) = P(X = x,\ Y = y) \\[4pt] p_{X,Y}(x, y) \ge 0 \\[4pt] \textstyle\sum_x \sum_y p_{X,Y}(x, y) = 1 \end{gathered}

One entry for every pair of values. The probability of any statement about both variables is the sum of the cells where it holds.

Three events read off the table:

$P(X = Y)$ : the diagonal, $\frac{1 + 2 + 1}{8} = \frac{1}{2}$ .
$P(X > Y)$ : the cells below the diagonal, $\frac{1 + 0 + 1}{8} = \frac{1}{4}$ .
$P(X + Y \ge 3)$ : the cells $(1, 2)$ , $(2, 1)$ and $(2, 2)$ , $\frac{3}{8}$ .

\frac{1}{8}

from fractions import Fraction from itertools import product import numpy as np p = [[Fraction(0)] * 3 for _ in range(3)] for c1, c2, c3 in product([0, 1], repeat=3): p[c1 + c2][c2 + c3] += Fraction(1, 8) print([[int(8 * v) for v in row] for row in p]) print(sum(p[i][i] for i in range(3)), sum(p[i][j] for i in range(3) for j in range(3) if i > j)) rng = np.random.default_rng(551) coins = rng.integers(0, 2, size=(100_000, 3)) X = coins[:, 0] + coins[:, 1] Y = coins[:, 1] + coins[:, 2] share = [[np.mean((X == i) & (Y == j)) for j in range(3)] for i in range(3)] print(np.round(8 * np.array(share), 3), round(np.mean(X == Y), 4))

⌘/Ctrl + Enter runs the cell

Marginal distributions

The distribution of $X$ alone sits in the table too. $X = 1$ happens in three cells, so $P(X = 1) = \frac{1}{8} + \frac{2}{8} + \frac{1}{8} = \frac{1}{2}$ .

Adding each row gives $p_X$ , and adding each column gives $p_Y$ . Written in the margins of the table, they are called the marginal distributions.

	$y = 0$	$y = 1$	$y = 2$	$p_X(x)$
$x = 0$	1/8	1/8	0	1/4
$x = 1$	1/8	2/8	1/8	1/2
$x = 2$	0	1/8	1/8	1/4
$p_Y(y)$	1/4	1/2	1/4	1

Both are binomial with $n = 2$ and $p = \frac{1}{2}$ (lesson 5.2.2), as two tosses each should be.

Rule: Marginal pmf

\begin{gathered} p_X(x) = \textstyle\sum_y p_{X,Y}(x, y) \\[4pt] p_Y(y) = \textstyle\sum_x p_{X,Y}(x, y) \end{gathered}

Add over every value of the other variable: across a row for $X$ , down a column for $Y$ . Each marginal adds up to 1.

The marginals do not fix the joint. Count $Y$ on two new coins instead, so the four tosses share nothing. The marginals stay $\frac{1}{4}, \frac{1}{2}, \frac{1}{4}$ , but every cell becomes a product: $P(X = 0, Y = 2) = \frac{1}{4} \cdot \frac{1}{4} = \frac{1}{16}$ instead of 0.

Then $P(X = Y) = \frac{3}{8}$ , not $\frac{1}{2}$ . Two variables need their joint distribution, not just one distribution each.

Conditional distributions

Suppose you learn that $X = 0$ : the first two tosses were tails. Only the row $x = 0$ is still possible, so the sample space shrinks to it, as in lesson 5.1.3. Its cells, $\frac{1}{8}$ , $\frac{1}{8}$ and 0, add up to $p_X(0) = \frac{1}{4}$ ; divided by that total, they become probabilities again: $\frac{1}{2}$ , $\frac{1}{2}$ and 0.

Rule: Conditional pmf

\begin{gathered} p_{Y \mid X}(y \mid x) = \frac{p_{X,Y}(x, y)}{p_X(x)} \\[4pt] \text{for } p_X(x) > 0 \end{gathered}

The row of $x$ divided by its own total. It is the distribution of $Y$ once you know that $X = x$ , and it adds up to 1 over $y$ . Given a value with $p_X(x) = 0$ , it is not defined.

Each conditional distribution has a mean, the conditional mean $\mathbb{E}[Y \mid X = x] = \sum_y y\,p_{Y \mid X}(y \mid x)$ (lesson 5.3.1):

given	$y = 0$	$y = 1$	$y = 2$	$\mathbb{E}[Y \mid X = x]$
$X = 0$	1/2	1/2	0	0.5
$X = 1$	1/4	1/2	1/4	1
$X = 2$	0	1/2	1/2	1.5

Knowing $X$ moves $Y$ : each extra head among the first two tosses raises the expected $Y$ by 0.5. Weighting the three conditional means by $p_X$ gives back $\mathbb{E}[Y] = \frac{1}{4}(0.5) + \frac{1}{2}(1) + \frac{1}{4}(1.5) = 1$ , the law of total probability of lesson 5.1.3 applied to means. Case study 5.5.5 uses conditional means of asset returns.

Conditioning the other way divides by a column total instead. $P(Y = 2 \mid X = 1) = \frac{1/8}{1/2} = \frac{1}{4}$ , but $P(X = 1 \mid Y = 2) = \frac{1/8}{1/4} = \frac{1}{2}$ .

x

from fractions import Fraction from itertools import product import numpy as np p = [[Fraction(0)] * 3 for _ in range(3)] for c1, c2, c3 in product([0, 1], repeat=3): p[c1 + c2][c2 + c3] += Fraction(1, 8) for x in range(3): row = [v / sum(p[x]) for v in p[x]] print(x, [str(v) for v in row], float(sum(y * row[y] for y in range(3)))) rng = np.random.default_rng(551) coins = rng.integers(0, 2, size=(100_000, 3)) X = coins[:, 0] + coins[:, 1] Y = coins[:, 1] + coins[:, 2] for x in range(3): given = Y[X == x] print(x, np.round([np.mean(given == y) for y in range(3)], 3), round(given.mean(), 3))

⌘/Ctrl + Enter runs the cell

Joint densities

Two continuous random variables have a joint density $f_{X,Y}(x, y)$ : a surface over the plane, never negative, with volume 1 under it. The probability that the pair lands in a region $R$ is the volume over $R$ , a double integral (lesson 4.7.1). A joint density is a probability per unit area: per unit of $x$ times unit of $y$ .

Case study 4.7.5 met one joint density, for two normal returns. Here is a simpler one: $f_{X,Y}(x, y) = x + y$ on the unit square $0 \le x \le 1$ , $0 \le y \le 1$ , and 0 outside it. Its volume is $\int_0^1 \int_0^1 (x + y)\,dy\,dx = \frac{1}{2} + \frac{1}{2} = 1$ .

$P(X < \frac{1}{2},\ Y < \frac{1}{2})$ , a square: $\int_0^{1/2} \int_0^{1/2} (x + y)\,dy\,dx = \frac{1}{8}$ .
$P(X + Y < 1)$ , the triangle below the line $y = 1 - x$ : $\int_0^1 \int_0^{1 - x} (x + y)\,dy\,dx = \frac{1}{3}$ .

The density is a height, not a probability: $f_{X,Y}(1, 1) = 2$ . As in lesson 5.4.1, the probability of any single point, or of a line, is 0.

Marginal density. A sum over $y$ becomes an integral over $y$ : $f_X(x) = \int_0^1 (x + y)\,dy = x + \frac{1}{2}$ on $[0, 1]$ . That is the area of the slice of the surface at $x$ (lesson 4.7.1). By symmetry $f_Y(y) = y + \frac{1}{2}$ .

Conditional density. Given $X = x$ , divide the slice at $x$ by its area, so that it has area 1:

f_{Y \mid X}(y \mid x) = \frac{f_{X,Y}(x, y)}{f_X(x)} = \frac{x + y}{x + \frac{1}{2}}

At $x = 0.25$ it runs from 0.3333 at $y = 0$ to 1.6667 at $y = 1$ , and its mean is $\mathbb{E}[Y \mid X = 0.25] = 0.6111$ . At $x = 0.75$ it is flatter, from 0.6 to 1.4, with mean 0.5667.

The density $x + y$ and two slices of it, each rescaled to area 1: the conditional densities of $Y$ . The slice at $x = 0.25$ tilts more, so it puts more weight on large $y$ and has the larger mean.

Rule: Joint, marginal and conditional densities

\begin{gathered} P\big((X, Y) \in R\big) = \textstyle\iint_R f_{X,Y}\,dA \\[4pt] f_X(x) = \textstyle\int f_{X,Y}(x, y)\,dy \\[4pt] f_{Y \mid X}(y \mid x) = \dfrac{f_{X,Y}(x, y)}{f_X(x)} \end{gathered}

The same three ideas as for a table, with sums replaced by integrals. The conditional density is defined where $f_X(x) > 0$ , even though $P(X = x)$ is 0.

Since $P(X = x) = 0$ , the conditional density cannot come from lesson 5.1.3's $P(A \mid B)$ directly. Think of it as conditioning on $X$ falling in a thin strip around $x$ , and letting the strip shrink. The simulation below does exactly that.

x = 0.25

import numpy as np from scipy.integrate import dblquad, quad f = lambda y, x: x + y print(round(dblquad(f, 0, 1, 0, 1)[0], 6), round(dblquad(f, 0, 0.5, 0, 0.5)[0], 6), round(dblquad(f, 0, 1, 0, lambda x: 1 - x)[0], 6)) print(round(quad(lambda y: y * (0.25 + y) / 0.75, 0, 1)[0], 4), round(quad(lambda y: y * (0.75 + y) / 1.25, 0, 1)[0], 4)) rng = np.random.default_rng(552) pts = rng.uniform(0, 1, size=(400_000, 3)) keep = 2 * pts[:, 2] < pts[:, 0] + pts[:, 1] # keep with probability (x + y) / 2 x, y = pts[keep, 0], pts[keep, 1] strip = (x > 0.2) & (x < 0.3) print(keep.sum(), round(np.mean(x + y < 1), 4), round(y[strip].mean(), 4))

⌘/Ctrl + Enter runs the cell

Independence

Lesson 5.1.3 called two events independent when $P(A \cap B) = P(A)\,P(B)$ . Two random variables are independent when this holds for every event about $X$ and every event about $Y$ . For a table, that means every cell equals the product of its two margins.

Rule: Independent random variables

\begin{gathered} p_{X,Y}(x, y) = p_X(x)\,p_Y(y) \\[4pt] f_{X,Y}(x, y) = f_X(x)\,f_Y(y) \\[4pt] \text{for every } x \text{ and } y \end{gathered}

The joint distribution is the product of the marginals. Then every conditional distribution of $Y$ equals its marginal: knowing $X$ tells you nothing about $Y$ . With $n$ variables, the joint is the product of all $n$ marginals.

Four checks:

The three coins: dependent. The cell $(0, 2)$ holds 0, but $p_X(0)\,p_Y(2) = \frac{1}{16}$ . One failed cell is enough. Five of the nine cells pass, the middle one among them, $\frac{2}{8} = \frac{1}{2} \cdot \frac{1}{2}$ , which proves nothing.
$x + y$ on the square: dependent. At $(0, 0)$ the density is 0, but $f_X(0)\,f_Y(0) = \frac{1}{4}$ . The figure shows it too: the conditional density of $Y$ changes with $x$ .
$4xy$ on the square: independent. It factors as $(2x)(2y)$ , and the marginals are $f_X(x) = 2x$ and $f_Y(y) = 2y$ .
2 on the triangle $0 < y < x < 1$ : dependent. The formula is a constant, which looks like a product, but the density is 0 off the triangle. The marginals are $f_X(x) = 2x$ and $f_Y(y) = 2(1 - y)$ , and at $(0.2, 0.8)$ their product is 0.16 where $f$ is 0. Knowing $X = 0.2$ forces $Y < 0.2$ .

The last check gives a quick test. If the region where the density is positive is not a rectangle, the variables are dependent, whatever the formula.

Independent normal draws

Case study 3.7.4 drew a table of standard normal numbers and said its columns were independent, "one tells you nothing about the others". Now that is precise. The three numbers in a row, $Z_1$ , $Z_2$ and $Z_3$ , have the joint density $\varphi(z_1)\,\varphi(z_2)\,\varphi(z_3)$ , with $\varphi$ the standard normal density.

So the conditional density of $Z_2$ given $Z_1 = z_1$ is $\varphi(z_2)$ , whatever $z_1$ is. In that table (seed 37), the 1,597 rows with $Z_1 > 1$ have $Z_2$ with mean $-0.006$ and standard deviation 0.996. All 10,000 rows give $-0.015$ and 0.993.

Case study 4.7.5's correlated returns are the opposite case. Their density is not a product. So the chance that both lose more than 10% was 0.0288, not the product of the single chances, 0.0145.

(0.2, 0.8)

from fractions import Fraction from itertools import product import numpy as np p = [[Fraction(0)] * 3 for _ in range(3)] for c1, c2, c3 in product([0, 1], repeat=3): p[c1 + c2][c2 + c3] += Fraction(1, 8) pX = [sum(row) for row in p] pY = [sum(p[i][j] for i in range(3)) for j in range(3)] print([[p[i][j] == pX[i] * pY[j] for j in range(3)] for i in range(3)]) print(round(2 * 0.2 * 2 * (1 - 0.8), 6)) Z = np.random.default_rng(37).standard_normal((10_000, 3)) for rows in (Z[:, 0] > 1, Z[:, 0] < -1, np.full(10_000, True)): print(rows.sum(), round(Z[rows, 1].mean(), 3), round(Z[rows, 1].std(ddof=1), 3))

⌘/Ctrl + Enter runs the cell

Lesson 5.5.2 measures how strongly two dependent variables move together. Case study 5.5.5 uses independence across months and days to scale risk over time.

Common mistakes

The marginals are not the whole story. The coin table and the four-coin table share their marginals, yet $P(X = Y)$ is $\frac{1}{2}$ in one and $\frac{3}{8}$ in the other.
Divide by the total of what you know. $p_{Y \mid X}(y \mid x)$ divides by the row total $p_X(x)$ . $P(Y = 2 \mid X = 1) = \frac{1}{4}$ and $P(X = 1 \mid Y = 2) = \frac{1}{2}$ are different numbers.
Independence needs every cell. Five of the coin table's nine cells equal their products, yet $X$ and $Y$ are dependent.
A density can exceed 1. $f_{X,Y}(1, 1) = 2$ is a height; only volumes are probabilities.
Look at the region, not only the formula. A constant density on a triangle gives dependent variables.
No conditioning on the impossible. Given a value with $p_X(x) = 0$ , the conditional distribution is not defined.

Joint, marginal and conditional distributions

0 of 12 correct

1.
In the coin table, find $P(X = 1, Y = 2)$
2.
In the coin table, find $P(X + Y \le 1)$
3.
A joint pmf has row $x = 0$ equal to $0.1, 0.3$ and row $x = 1$ equal to $0.2, 0.4$ , for $y = 0, 1$ . Find $p_Y(1)$
4.
For the same table, find $P(Y = 1 \mid X = 0)$
5.
For the same table, find $\mathbb{E}[Y \mid X = 1]$
6.
In the coin table, find $P(X = 2 \mid Y = 1)$
7.
For the density $x + y$ on the unit square, find the marginal density $f_X(0.3)$
8.
For the density $x + y$ on the unit square, find $P(X < \frac{1}{2})$
9.
For the density $x + y$ on the unit square, find the conditional density $f_{Y \mid X}(1 \mid 0)$
10.
$X$ and $Y$ are independent with $P(X = 1) = 0.3$ and $P(Y = 2) = 0.5$ . Find $P(X = 1, Y = 2)$
11.
$f(x, y) = 6xy^2$ on the unit square. Find the marginal density $f_Y(0.5)$
12.
$f(x, y) = 2$ on the triangle $0 < y < x < 1$ . Are $X$ and $Y$ independent? Type 1 for yes, 0 for no

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

p_{X,Y}(x, y) = P(X = x, Y = y)

Joint, marginal and conditional distributions

Two random variables on one experiment

Rule: Joint pmf

Marginal distributions

Rule: Marginal pmf

Conditional distributions

Rule: Conditional pmf

Joint densities

Rule: Joint, marginal and conditional densities

Independence

Rule: Independent random variables

Independent normal draws

Common mistakes

Joint, marginal and conditional distributions

Summary card

Free further reading