5.5.2 Covariance, correlation and independence

In this lesson

The mean of a function of two random variables, and $\mathbb{E}[XY]$
Covariance: what its sign says, its shortcut $\mathbb{E}[XY] - \mu_X\mu_Y$ , and its units
Correlation $\rho$ : covariance without units, always between $-1$ and 1
The rules for $\operatorname{Var}(X + Y)$ , $\operatorname{Var}(X - Y)$ and $\operatorname{Var}(aX + bY)$
Independent variables are uncorrelated, but uncorrelated variables need not be independent
Sample covariance, standard deviation and correlation, dividing by $n - 1$
The covariance matrix of a random vector, $\mathbb{E}[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^\top]$

You need joint distributions and independence from lesson 5.5.1, and expectation and LOTUS from lesson 5.3.1. You also need variance and linearity of expectation from lesson 5.3.2. Case study 3.1.4's correlation as a cosine and case study 3.2.5's covariance matrix come back with their numbers.

The mean of a function of two variables

Lesson 5.3.1 found $\mathbb{E}[g(X)]$ by weighting each value $g(x)$ by its probability. With two variables, weight each $g(x, y)$ by the joint probability of its cell (lesson 5.5.1):

Rule: Expectation of g(X, Y)

\begin{gathered} \mathbb{E}[g(X, Y)] \\ = \textstyle\sum_x\sum_y g(x, y)\,p_{X,Y}(x, y) \\[4pt] = \textstyle\iint g(x, y)\,f_{X,Y}(x, y)\,dA \end{gathered}

The sum for a joint pmf, the double integral for a joint density. No distribution of $g(X, Y)$ is needed.

Take lesson 5.5.1's three coins: $X$ counts heads in tosses 1 and 2, $Y$ in tosses 2 and 3. Only the cells where both are positive add to $\mathbb{E}[XY]$ . The cell $(1, 1)$ has $xy = 1$ and two outcomes; $(1, 2)$ , $(2, 1)$ and $(2, 2)$ have $xy = 2$ , 2 and 4 and one outcome each:

\begin{aligned} \mathbb{E}[XY] &= \tfrac{1}{8}(1 \cdot 2 + 2 + 2 + 4) \\ &= \tfrac{10}{8} = \tfrac{5}{4} \end{aligned}

With $g(x, y) = x + y$ the same sum gives $\mathbb{E}[X + Y] = 2 = \mathbb{E}[X] + \mathbb{E}[Y]$ , the linearity of lesson 5.3.2. Products behave differently: $\mathbb{E}[XY] = 1.25$ , but $\mathbb{E}[X]\,\mathbb{E}[Y] = 1$ .

For independent $X$ and $Y$ the product does split, $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$ , because the joint is the product of the marginals and the double sum factors. More generally $\mathbb{E}[g(X)\,h(Y)] = \mathbb{E}[g(X)]\,\mathbb{E}[h(Y)]$ .

Covariance

Write $\mu_X = \mathbb{E}[X]$ and $\mu_Y = \mathbb{E}[Y]$ for the two means.

When $X$ and $Y$ tend to be above their means together and below them together, the product $(X - \mu_X)(Y - \mu_Y)$ is mostly positive. When one tends to be high while the other is low, it is mostly negative. Its mean measures which pattern wins.

Rule: Covariance

\begin{gathered} \operatorname{Cov}(X, Y) \\ = \mathbb{E}\big[(X - \mu_X)(Y - \mu_Y)\big] \\[4pt] = \mathbb{E}[XY] - \mu_X\,\mu_Y \end{gathered}

Positive: the two move together; negative: they move oppositely. Its units are $X$ 's units times $Y$ 's. $\operatorname{Cov}(X, X) = \operatorname{Var}(X)$ , and $\operatorname{Cov}(Y, X) = \operatorname{Cov}(X, Y)$ .

The coins. Here $\mu_X = \mu_Y = 1$ . The product $(x - 1)(y - 1)$ is 1 in the cells $(0, 0)$ and $(2, 2)$ , each with probability $\frac{1}{8}$ .

In the cells $(0, 2)$ and $(2, 0)$ , where it would be $-1$ , the probability is 0, and every other cell gives 0. So $\operatorname{Cov}(X, Y) = \frac{2}{8} = \frac{1}{4}$ , and the shortcut agrees: $\frac{5}{4} - 1 \cdot 1 = \frac{1}{4}$ .

A density. For lesson 5.5.1's $f_{X,Y}(x, y) = x + y$ on the unit square, double integrals give $\mu_X = \mu_Y = \frac{7}{12}$ and $\mathbb{E}[XY] = \frac{1}{3}$ . So $\operatorname{Cov}(X, Y) = \frac{1}{3} - \frac{49}{144} = -\frac{1}{144}$ , about $-0.00694$ . It is slightly negative, which matches lesson 5.5.1's finding: the conditional mean of $Y$ fell from 0.6111 at $x = 0.25$ to 0.5667 at $x = 0.75$ .

Each dot adds the product of its two deviations from the means: positive in the shaded quarters, negative in the others. Covariance is their average. In the second picture about half the dots are shaded and the products cancel, yet $Y$ depends completely on $X$ .

g(x, y)

from fractions import Fraction from itertools import product import numpy as np p = {} for c1, c2, c3 in product([0, 1], repeat=3): cell = (c1 + c2, c2 + c3) p[cell] = p.get(cell, Fraction(0)) + Fraction(1, 8) def E(g): return sum(g(x, y) * q for (x, y), q in p.items()) mx, my = E(lambda x, y: x), E(lambda x, y: y) print(E(lambda x, y: x * y), E(lambda x, y: (x - mx) * (y - my)), E(lambda x, y: x * y) - mx * my, E(lambda x, y: x + y)) coins = np.random.default_rng(553).integers(0, 2, size=(100_000, 3)) X, Y = coins[:, 0] + coins[:, 1], coins[:, 1] + coins[:, 2] print(round(np.cov(X, Y)[0, 1], 4))

⌘/Ctrl + Enter runs the cell

Correlation

A covariance depends on units: measure $X$ in cents instead of dollars and it grows 100 times. Dividing by both standard deviations, $\sigma_X = \sqrt{\operatorname{Var}(X)}$ and $\sigma_Y$ , removes the units.

Rule: Correlation

\begin{gathered} \rho = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\,\sigma_Y}, \quad -1 \le \rho \le 1 \\[4pt] \operatorname{Cov}(X, Y) = \rho\,\sigma_X\,\sigma_Y \end{gathered}

$\rho$ (rho) has no units. It is 1 or $-1$ exactly when $Y = a + bX$ for some numbers $a$ and $b \neq 0$ , with the sign of $b$ . It is undefined when either variable is constant.

For the coins, $\operatorname{Var}(X) = \operatorname{Var}(Y) = \frac{1}{2}$ (two tosses, lesson 5.3.2), so $\rho = \frac{1/4}{1/2} = 0.5$ . For the density $x + y$ , $\operatorname{Var}(X) = \frac{5}{12} - \frac{49}{144} = \frac{11}{144}$ , so $\rho = \frac{-1/144}{11/144} = -\frac{1}{11} \approx -0.0909$ : a weak link.

Case study 3.1.4 met $\rho$ as the cosine of an angle between centred return vectors, which is why it lies between $-1$ and 1. Case study 3.2.5 took the covariance of two assets to be $\rho_{ij}\sigma_i\sigma_j$ . That is the second line of the rule: covariance defined from probability, then written with the correlation.

\rho

import numpy as np from scipy.integrate import dblquad def E(g): return dblquad(lambda y, x: g(x, y) * (x + y), 0, 1, 0, 1)[0] mx, ex2, exy = E(lambda x, y: x), E(lambda x, y: x * x), E(lambda x, y: x * y) cov, var = exy - mx**2, ex2 - mx**2 print(round(mx, 6), round(exy, 6), round(cov, 6), round(var, 6)) print(0.25 / 0.5, round(cov / var, 4), round(-1 / 11, 4)) pts = np.random.default_rng(552).uniform(0, 1, size=(400_000, 3)) keep = 2 * pts[:, 2] < pts[:, 0] + pts[:, 1] x, y = pts[keep, 0], pts[keep, 1] print(round(np.cov(x, y)[0, 1], 4), round(np.corrcoef(x, y)[0, 1], 4))

⌘/Ctrl + Enter runs the cell

Scaling changes covariance but not correlation. $\operatorname{Cov}(aX + b,\ cY + d) = ac\,\operatorname{Cov}(X, Y)$ , while the correlation of $aX + b$ and $cY + d$ is $\rho$ when $ac > 0$ and $-\rho$ when $ac < 0$ . Adding a constant changes neither.

Variances of sums

Covariance obeys the same algebra as multiplying out brackets: it is bilinear. Constants come out of each slot, sums split, and a constant variable has covariance 0 with anything:

\begin{gathered} \operatorname{Cov}(aX + bY,\ W) \\ = a\operatorname{Cov}(X, W) + b\operatorname{Cov}(Y, W) \end{gathered}

Since $\operatorname{Var}(X + Y) = \operatorname{Cov}(X + Y,\ X + Y)$ , multiplying out gives four terms: $\operatorname{Var}(X)$ , $\operatorname{Var}(Y)$ and $\operatorname{Cov}(X, Y)$ twice.

Rule: Variance of a sum

\begin{gathered} \operatorname{Var}(X \pm Y) = \operatorname{Var}(X) \\ + \operatorname{Var}(Y) \pm 2\operatorname{Cov}(X, Y) \\[8pt] \operatorname{Var}(aX + bY) = a^2\operatorname{Var}(X) \\ + b^2\operatorname{Var}(Y) + 2ab\operatorname{Cov}(X, Y) \end{gathered}

Read $\pm$ as $+$ for $X + Y$ and $-$ for $X - Y$ . Variances never subtract: $\operatorname{Var}(-Y) = \operatorname{Var}(Y)$ , so only the covariance term carries the sign. With $n$ variables, $\operatorname{Var}(X_1 + \cdots + X_n)$ is the sum of all $n$ variances plus twice every covariance $\operatorname{Cov}(X_i, X_j)$ with $i < j$ .

Check it on the coins, two ways.

From the table. $X + Y$ takes the values 0 to 4 with probabilities $\frac{1}{8}, \frac{1}{4}, \frac{1}{4}, \frac{1}{4}, \frac{1}{8}$ , so its variance is $\frac{3}{2}$ . The rule says $\frac{1}{2} + \frac{1}{2} + 2 \cdot \frac{1}{4} = \frac{3}{2}$ . For $X - Y$ it says $\frac{1}{2} + \frac{1}{2} - \frac{1}{2} = \frac{1}{2}$ , and the table agrees.

By algebra. Write the coins as $C_1$ , $C_2$ and $C_3$ , each 1 for heads and 0 for tails, independent, each with variance $\frac{1}{4}$ . Then $X = C_1 + C_2$ and $Y = C_2 + C_3$ .

By bilinearity, $\operatorname{Cov}(X, Y)$ splits into four covariances, and only $\operatorname{Cov}(C_2, C_2) = \frac{1}{4}$ is not 0. The shared coin is the whole covariance.

Then $X + Y = C_1 + 2C_2 + C_3$ has variance $\frac{1}{4} + 4 \cdot \frac{1}{4} + \frac{1}{4} = \frac{3}{2}$ , and $X - Y = C_1 - C_3$ has variance $\frac{1}{2}$ : the shared coin cancels.

Lesson 5.3.2's top face plus bottom face of one die is the opposite case. The bottom face is $7 - X$ , so $\operatorname{Cov}(X, 7 - X) = -\operatorname{Var}(X)$ , and the rule gives $\operatorname{Var}(X) + \operatorname{Var}(X) - 2\operatorname{Var}(X) = 0$ : the total is always 7.

With every covariance 0, the variances simply add. That is how lesson 5.3.2 found the binomial variance $np(1 - p)$ , a sum of $n$ independent Bernoulli variables. Case study 5.5.5 applies the rule with many variables to a portfolio, and to returns added over time.

X + Y

from fractions import Fraction from itertools import product import numpy as np p = {} for c1, c2, c3 in product([0, 1], repeat=3): cell = (c1 + c2, c2 + c3) p[cell] = p.get(cell, Fraction(0)) + Fraction(1, 8) total = {} for (x, y), q in p.items(): total[x + y] = total.get(x + y, Fraction(0)) + q print({s: str(q) for s, q in sorted(total.items())}) print(sum((s - 2)**2 * q for s, q in total.items()), sum((x - y)**2 * q for (x, y), q in p.items())) var, cov = Fraction(1, 2), Fraction(1, 4) print(var + var + 2 * cov, var + var - 2 * cov) coins = np.random.default_rng(553).integers(0, 2, size=(100_000, 3)) X, Y = coins[:, 0] + coins[:, 1], coins[:, 1] + coins[:, 2] print(round(np.var(X + Y, ddof=1), 4), round(np.var(X - Y, ddof=1), 4))

⌘/Ctrl + Enter runs the cell

Independent means uncorrelated, not the reverse

If $X$ and $Y$ are independent, $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$ , so $\operatorname{Cov}(X, Y) = 0$ and the variances add. The reverse fails, and case study 3.1.4 already had an example.

Its four months of returns were $(1, 2, 3, 4)$ and $(4, 2, 2, 4)$ , in percent. As random variables: let $X$ be 1, 2, 3 or 4, each with probability $\frac{1}{4}$ . Let $Y = 4$ when $X$ is 1 or 4, and $Y = 2$ when $X$ is 2 or 3.

Then $\mathbb{E}[XY] = \frac{4 + 4 + 6 + 16}{4} = 7.5$ and $\mathbb{E}[X]\,\mathbb{E}[Y] = 2.5 \times 3 = 7.5$ , so the covariance is 0.

Yet $Y$ is fixed by $X$ . $P(Y = 2) = \frac{1}{2}$ , but $P(Y = 2 \mid X = 1) = 0$ , so they are dependent (lesson 5.5.1).

The same happens with $X$ standard normal and $Y = X^2$ , the second picture above. Large values of $Y$ come with large $\lvert X \rvert$ of either sign, and the positive and negative products cancel.

Rule: Independence and correlation

\begin{gathered} \text{independent} \\ \implies \operatorname{Cov}(X, Y) = 0 \\[6pt] \operatorname{Cov}(X, Y) = 0 \\ \not\Longrightarrow \text{independent} \end{gathered}

Covariance measures only how closely $Y$ follows a straight line in $X$ . A curved link, or one that changes sign, can have covariance 0. Lesson 5.5.3 shows that for jointly normal variables zero correlation does mean independence.

\mathbb{E}[XY] - \mathbb{E}[X]\,\mathbb{E}[Y] = 0

import numpy as np x = np.array([1, 2, 3, 4]) y = np.array([4, 2, 2, 4]) print((x * y).mean(), x.mean() * y.mean(), (x * y).mean() - x.mean() * y.mean()) z = np.random.default_rng(554).standard_normal(100_000) print(round(np.corrcoef(z, z**2)[0, 1], 4), round(np.corrcoef(np.abs(z), z**2)[0, 1], 4))

⌘/Ctrl + Enter runs the cell

Sample covariance, standard deviation and correlation

With data instead of a distribution, each expectation becomes an average over the sample. For $n$ pairs $(x_t, y_t)$ , write $\bar{x}$ and $\bar{y}$ for the sample means. The usual estimates divide the sums by $n - 1$ rather than $n$ ; lesson 5.8.1 explains why.

Rule: Sample versions

\begin{gathered} s_x^2 = \frac{1}{n - 1}\textstyle\sum_t (x_t - \bar{x})^2 \\[4pt] s_{xy} = \frac{1}{n - 1}\textstyle\sum_t (x_t - \bar{x})(y_t - \bar{y}) \\[4pt] r = \frac{s_{xy}}{s_x\,s_y} \end{gathered}

$s_x^2$ is the sample variance and its square root $s_x$ the sample standard deviation; $s_{xy}$ is the sample covariance and $r$ the sample correlation. The factors $n - 1$ cancel in $r$ .

Case study 3.1.4's ten months of returns for stocks X and Y give, in decimals:

	value	as in case study 3.1.4
$s_{xy}$	0.000679	$\tilde{\mathbf{x}} \cdot \tilde{\mathbf{y}}$ divided by 9
$s_x$	0.050284	$\lVert \tilde{\mathbf{x}} \rVert$ divided by 3
$s_y$	0.022088	$\lVert \tilde{\mathbf{y}} \rVert$ divided by 3
$r$	0.6113	the cosine of 52.32°

These are the sample standard deviation and covariance that case study 3.1.4 promised to define. The sample correlation is the cosine it computed.

In NumPy, np.cov(x, y) and np.corrcoef(x, y) give the sample covariance matrix and correlation. np.var and np.std divide by $n$ unless you pass ddof=1: for X's returns, 0.002276 instead of 0.002528.

s_{xy}

import numpy as np rng = np.random.default_rng(449) market = rng.normal(0.008, 0.04, 10) x = np.round(market + rng.normal(0.002, 0.025, 10), 3) y = np.round(0.6 * market + rng.normal(0.003, 0.02, 10), 3) n = len(x) xc, yc = x - x.mean(), y - y.mean() s_xy = xc @ yc / (n - 1) s_x, s_y = np.sqrt(xc @ xc / (n - 1)), np.sqrt(yc @ yc / (n - 1)) print(round(s_xy, 6), round(s_x, 6), round(s_y, 6), round(s_xy / (s_x * s_y), 4)) print(round(np.cov(x, y)[0, 1], 6), round(np.std(x, ddof=1), 6), round(np.corrcoef(x, y)[0, 1], 4)) print(round(np.var(x), 6), round(np.var(x, ddof=1), 6), round(np.cov(100 * x, 100 * y)[0, 1], 4), round(np.corrcoef(100 * x, 100 * y)[0, 1], 4))

⌘/Ctrl + Enter runs the cell

The covariance matrix

Several random variables $X_1, \ldots, X_n$ form a random vector $\mathbf{X}$ , a column written in bold capitals. Its mean is the vector of means, $\boldsymbol{\mu} = \mathbb{E}[\mathbf{X}]$ , taken entry by entry.

The product $(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^\top$ is an $n \times n$ matrix whose entry $(i, j)$ is $(X_i - \mu_i)(X_j - \mu_j)$ . Its expectation, entry by entry, holds every covariance.

Rule: Covariance matrix

\begin{gathered} \Sigma = \mathbb{E}\big[(\mathbf{X} - \boldsymbol{\mu})(\mathbf{X} - \boldsymbol{\mu})^\top\big] \\[4pt] \Sigma_{ij} = \operatorname{Cov}(X_i, X_j) = \rho_{ij}\,\sigma_i\,\sigma_j \end{gathered}

The diagonal holds the variances $\sigma_i^2$ ; off the diagonal are the covariances. $\Sigma$ is symmetric, because $\operatorname{Cov}(X_i, X_j) = \operatorname{Cov}(X_j, X_i)$ .

For the coins, with $\mathbf{X} = (X, Y)$ :

\Sigma = \begin{pmatrix} 1/2 & 1/4 \\ 1/4 & 1/2 \end{pmatrix}

Case study 3.2.5 built the three assets' $\Sigma$ as $DRD$ from volatilities and correlations. By the rule above, that is this matrix for their random returns.

Lesson 3.7.1 showed that a covariance matrix is positive semidefinite. np.cov(data, rowvar=False) is the sample version $S$ (case study 3.2.5), with one row per observation.

p \cdot (\mathbf{x} - \boldsymbol{\mu})(\mathbf{x} - \boldsymbol{\mu})^\top

from fractions import Fraction from itertools import product import numpy as np p = {} for c1, c2, c3 in product([0, 1], repeat=3): cell = (c1 + c2, c2 + c3) p[cell] = p.get(cell, Fraction(0)) + Fraction(1, 8) mu = np.array([1, 1]) Sigma = sum(float(q) * np.outer(np.array(cell) - mu, np.array(cell) - mu) for cell, q in p.items()) print(Sigma) coins = np.random.default_rng(553).integers(0, 2, size=(100_000, 3)) data = np.column_stack([coins[:, 0] + coins[:, 1], coins[:, 1] + coins[:, 2]]) print(np.round(np.cov(data, rowvar=False), 4))

⌘/Ctrl + Enter runs the cell

Lesson 5.5.3 builds random vectors with a chosen $\Sigma$ , and case study 5.5.5 uses these rules to derive the variance of a portfolio.

Common mistakes

Variances add only without covariance. For the coins $\operatorname{Var}(X + Y) = \frac{3}{2}$ , not $\frac{1}{2} + \frac{1}{2} = 1$ .
Variances never subtract. $\operatorname{Var}(X - Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) - 2\operatorname{Cov}(X, Y)$ , here $\frac{1}{2}$ , not $\frac{1}{2} - \frac{1}{2} = 0$ .
Constants come out squared. $\operatorname{Var}(2X) = 4\operatorname{Var}(X)$ , while $\operatorname{Cov}(2X, Y) = 2\operatorname{Cov}(X, Y)$ .
Zero covariance is not independence. $X$ and $X^2$ have covariance 0 when $X$ is standard normal.
Compare correlations, not covariances. Percent units multiply a covariance by 10,000 and leave the correlation alone.
Check the divisor. np.var and np.std divide by $n$ by default; np.cov divides by $n - 1$ .

Covariance, correlation and independence

0 of 12 correct

1.
$X$ and $Y$ each take the values 0 and 1. $P(X = 1, Y = 1) = 0.4$ , $P(X = 1) = 0.5$ and $P(Y = 1) = 0.6$ . Find $\mathbb{E}[XY]$
2.
For the same pair, find $\operatorname{Cov}(X, Y)$
3.
$\operatorname{Cov}(X, Y) = 3$ , $\operatorname{Var}(X) = 4$ and $\operatorname{Var}(Y) = 9$ . Find $\rho$
4.
$\rho = -0.4$ , $\sigma_X = 5$ and $\sigma_Y = 2$ . Find $\operatorname{Cov}(X, Y)$
5.
$\operatorname{Var}(X) = 4$ , $\operatorname{Var}(Y) = 9$ and $\operatorname{Cov}(X, Y) = 3$ . Find $\operatorname{Var}(X + Y)$
6.
For the same pair, find $\operatorname{Var}(X - Y)$
7.
For the same pair, find $\operatorname{Var}(2X - 3Y)$
8.
$\operatorname{Cov}(X, Y) = 1.5$ . Find $\operatorname{Cov}(3X + 1,\ -2Y + 7)$
9.
Two fair dice are rolled independently; each has variance $\frac{35}{12}$ . Find the variance of their total
10.
$X$ is $-1$ , 0 or 1, each with probability $\frac{1}{3}$ , and $Y = X^2$ . Find $\operatorname{Cov}(X, Y)$
11.
Find the sample covariance of the pairs $(1, 2)$ , $(2, 4)$ and $(3, 9)$ , dividing by $n - 1$
12.
Find the correlation of $X$ and $5 - 3X$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\mathbb{E}[g(X, Y)] = \sum\sum g\,p_{X,Y}

Covariance, correlation and independence

The mean of a function of two variables

Rule: Expectation of g(X, Y)

Covariance

Rule: Covariance

Correlation

Rule: Correlation

Variances of sums

Rule: Variance of a sum

Independent means uncorrelated, not the reverse

Rule: Independence and correlation

Sample covariance, standard deviation and correlation

Rule: Sample versions

The covariance matrix

Rule: Covariance matrix

Common mistakes

Covariance, correlation and independence

Summary card

Free further reading