5.10.1 Simple linear regression

In this lesson

The model $y = \beta_0 + \beta_1x + \varepsilon$ : a straight line for the mean of $y$ , plus an error
The least-squares estimates of lesson 3.5.3, written with the sample covariance and variance of lesson 5.5.2
Fitted values and residuals, and the residual standard deviation $s$
$R^2$ : the share of the variation in $y$ that the line accounts for
How far the estimated slope can be from the true one, by simulation and by a formula
Why regressing $y$ on $x$ is not regressing $x$ on $y$
Checking every step in Python

You need least squares from lesson 3.5.3, and the sample covariance, variance and correlation from lesson 5.5.2. Estimators and their spread come from lessons 5.8.1 and 5.8.2, and case study 5.5.5's line of conditional means comes back.

The model

A regression describes how the mean of one variable changes with another. The variable to explain, $y$ , is the response; the one it is explained by, $x$ , is the regressor. Simple linear regression assumes that the mean of $y$ is a straight line in $x$ , and that each observation scatters around that line.

Rule: Simple linear regression

\begin{gathered} y_i = \beta_0 + \beta_1 x_i + \varepsilon_i \\[4pt] \mathbb{E}[\varepsilon_i] = 0, \quad \operatorname{Var}(\varepsilon_i) = \sigma^2 \end{gathered}

There are $n$ observations $(x_i, y_i)$ , $i = 1, \ldots, n$ . $\beta_0$ is the intercept and $\beta_1$ the slope, in units of $y$ per unit of $x$ .

The error $\varepsilon_i$ (epsilon) is random, with mean 0 and the same standard deviation $\sigma$ at every $x$ . So $\mathbb{E}[y_i] = \beta_0 + \beta_1x_i$ : the line is the mean of $y$ at each $x$ .

$\beta_0$ , $\beta_1$ and $\sigma$ are unknown numbers. Data give estimates of them, written with hats, $\hat{\beta}_0$ and $\hat{\beta}_1$ , as for the estimator $\hat{\theta}$ of lesson 5.8.1. Lesson 5.10.3 lists the assumptions in full and says what happens when they fail.

Lesson 3.5.3 made 20 points from the line $y = 2 + 0.5x$ , adding errors of standard deviation 1. That data follows this model with $\beta_0 = 2$ , $\beta_1 = 0.5$ and $\sigma = 1$ . These are known only because the data was invented with them.

Case study 5.5.5 met the model too. For two jointly normal variables, $\mathbb{E}[Y \mid X = x]$ is a straight line with slope $\rho\frac{\sigma_Y}{\sigma_X}$ : 0.6944 for its returns of B and C. The spread around it is the same at every $x$ , so the pair follows this model exactly: its regression line is the line of conditional means.

The least-squares estimates

Least squares (lesson 3.5.3) picks the line $b_0 + b_1x$ that makes the sum of squared vertical misses as small as possible:

\operatorname{RSS}(b_0, b_1) = \sum_{i=1}^n \big(y_i - b_0 - b_1x_i\big)^2

Lesson 3.5.3 found the best line from the normal equations. Lesson 4.4.1 set the gradient of RSS to zero, which gives the same two equations. Their solution reads best in the sample statistics of lesson 5.5.2.

Rule: Least-squares estimates

\begin{gathered} \hat{\beta}_1 = \frac{s_{xy}}{s_x^2} \\[6pt] = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \\[6pt] \hat{\beta}_0 = \bar{y} - \hat{\beta}_1\bar{x} \end{gathered}

$s_{xy}$ is the sample covariance and $s_x^2$ the sample variance of the $x_i$ ; their divisors $n - 1$ cancel. The line passes through the mean point $(\bar{x}, \bar{y})$ . Since $s_{xy} = r\,s_xs_y$ , the slope is also $\hat{\beta}_1 = r\frac{s_y}{s_x}$ , with the sign of the correlation $r$ .

A worked example

Lesson 1.2.1 gave np.polyfit four points that are not quite on a line, and it returned $y = 1.94x + 1.09$ . Here is how. The means are $\bar{x} = 1.5$ and $\bar{y} = 4$ .

$x_i$	0	1	2	3	sum
$y_i$	1.1	2.9	5.2	6.8
$x_i - \bar{x}$	$-1.5$	$-0.5$	0.5	1.5	0
$y_i - \bar{y}$	$-2.9$	$-1.1$	1.2	2.8	0
product	4.35	0.55	0.6	4.2	9.7
$(x_i - \bar{x})^2$	2.25	0.25	0.25	2.25	5

So $\hat{\beta}_1 = \frac{9.7}{5} = 1.94$ and $\hat{\beta}_0 = 4 - 1.94 \times 1.5 = 1.09$ . In sample statistics, $s_{xy} = \frac{9.7}{3} = 3.2333$ and $s_x^2 = \frac{5}{3} = 1.6667$ , whose ratio is again 1.94.

The noisy points of lesson 3.5.3 give $s_{xy} = 15.997$ and $s_x^2 = 35$ , so $\hat{\beta}_1 = 0.4571$ . With $\bar{x} = 10.5$ and $\bar{y} = 7.2152$ , the intercept is $\hat{\beta}_0 = 2.4162$ : the same line that lstsq found there.

Every sign, and where it breaks

$s_{xy} > 0$ : the line rises; $s_{xy} < 0$ : it falls.
$s_{xy} = 0$ : the line is flat at $\bar{y}$ . Knowing $x$ does not change the best straight-line guess of $y$ .
Every $x_i$ the same: $s_x^2 = 0$ and there is no slope. A best line would be vertical, as lesson 3.5.3 found.
Two points: the line goes through both, and every miss is 0.

Try it fit(x, y) is the rule: the slope from the sample covariance and variance, then the intercept through the mean point. The cell fits lesson 1.2.1's four points and checks them against np.polyfit, which lists the slope first. Then it rebuilds lesson 3.5.3's noisy points from their seed and fits them.

import numpy as np def fit(x, y): b1 = np.cov(x, y)[0, 1] / np.var(x, ddof=1) return y.mean() - b1 * x.mean(), b1 x = np.array([0.0, 1.0, 2.0, 3.0]) y = np.array([1.1, 2.9, 5.2, 6.8]) print(np.round(fit(x, y), 4), np.round(np.polyfit(x, y, 1), 4)) rng = np.random.default_rng(353) x = np.arange(1, 21, dtype=float) y = 2 + 0.5 * x + rng.normal(0, 1, 20) print(round(np.cov(x, y)[0, 1], 4), np.var(x, ddof=1), round(y.mean(), 4)) print(np.round(fit(x, y), 4))

⌘/Ctrl + Enter runs the cell

Fitted values and residuals

The fitted value at $x_i$ is the line's height there, $\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1x_i$ . The residual is what the line misses, $e_i = y_i - \hat{y}_i$ , a vertical gap as in lesson 3.5.3. For the four points:

$x_i$	0	1	2	3
$y_i$	1.1	2.9	5.2	6.8
$\hat{y}_i = 1.09 + 1.94x_i$	1.09	3.03	4.97	6.91
$e_i$	0.01	$-0.13$	0.23	$-0.11$

The residuals add up to 0, and so do the products $x_ie_i$ . That is lesson 3.5.3's right angle: the residual vector is orthogonal to the column of ones and to the column of $x$ values. The sum of squared residuals is $\operatorname{RSS} = 0.082$ .

Errors are not residuals

The error $\varepsilon_i = y_i - \beta_0 - \beta_1x_i$ is the gap to the true line, which no one sees. The residual $e_i$ is the gap to the fitted line, which you can compute. They differ because the fitted line differs from the true one.

The same 20 points measured from two lines. The errors are gaps to the true line, known here only because the data was invented; the residuals are gaps to the fitted line. Least squares makes the squared residuals add up to less than the squared errors.

For lesson 3.5.3's points, the squared errors add up to 25.3502 and the squared residuals to 24.0997. The residuals are smaller because least squares chose the line that minimises their sum, against every other line, the true one included. So the residuals understate the errors a little, and an estimate of $\sigma$ divides by $n - 2$ to make up for it.

Rule: Residual standard deviation

s = \sqrt{\frac{\sum_i e_i^2}{n - 2}}

$s$ estimates $\sigma$ , the typical size of an error, in the units of $y$ . The divisor is $n - 2$ because the line used two estimates, $\hat{\beta}_0$ and $\hat{\beta}_1$ . The sample variance of lesson 5.5.2 divided by $n - 1$ for its one estimate, the mean; lesson 5.8.1 explains why such corrections work.

For the four points, $s = \sqrt{0.082 / 2} = 0.2025$ . For lesson 3.5.3's 20 points, $s = 1.1571$ , against the true $\sigma = 1$ . With two points, $n - 2 = 0$ and there is nothing left to estimate $\sigma$ from.

s

import numpy as np x = np.array([0.0, 1.0, 2.0, 3.0]) y = np.array([1.1, 2.9, 5.2, 6.8]) b1 = np.cov(x, y)[0, 1] / np.var(x, ddof=1) b0 = y.mean() - b1 * x.mean() e = y - (b0 + b1 * x) print(np.round(b0 + b1 * x, 4), np.round(e, 4)) print(round(e.sum(), 10) + 0.0, round(x @ e, 10) + 0.0, round(e @ e, 4), round(np.sqrt(e @ e / 2), 4)) rng = np.random.default_rng(353) x = np.arange(1, 21, dtype=float) eps = rng.normal(0, 1, 20) y = 2 + 0.5 * x + eps b1 = np.cov(x, y)[0, 1] / np.var(x, ddof=1) b0 = y.mean() - b1 * x.mean() e = y - (b0 + b1 * x) print(round(eps @ eps, 4), round(e @ e, 4), round(np.sqrt(e @ e / 18), 4))

⌘/Ctrl + Enter runs the cell

R²: how much the line explains

The total sum of squares $\operatorname{TSS} = \sum_i (y_i - \bar{y})^2$ measures how much $y$ varies around its mean. RSS is what is left after the line. The share the line accounts for is $R^2$ .

Rule: R squared

\begin{gathered} R^2 = 1 - \frac{\operatorname{RSS}}{\operatorname{TSS}} \\[4pt] R^2 = r^2 \text{ for one regressor} \end{gathered}

$R^2$ lies between 0 and 1 and has no units. It is 1 when every point is on the line and 0 when the fitted line is flat. With one regressor it is the square of the sample correlation $r$ of lesson 5.5.2.

For the four points, $\operatorname{TSS} = 18.9$ and $\operatorname{RSS} = 0.082$ , so $R^2 = 0.9957$ ; the correlation is $r = 0.9978$ , whose square is the same. For lesson 3.5.3's 20 points, $R^2 = 0.8522$ . Case study 3.5.5 read the same number as the share of a stock's variance that a hedge removes.

$R^2$ measures how tightly the points hug the line, not how steep the line is. Two thousand points with a true slope of 0.1 and errors of size 1 give $R^2 = 0.0094$ . Yet the estimated slope is 0.0992, close to the truth: a small $R^2$ does not mean the slope is 0.

R^2

import numpy as np x = np.array([0.0, 1.0, 2.0, 3.0]) y = np.array([1.1, 2.9, 5.2, 6.8]) b1 = np.cov(x, y)[0, 1] / np.var(x, ddof=1) e = y - (y.mean() - b1 * x.mean() + b1 * x) tss = ((y - y.mean()) ** 2).sum() r = np.corrcoef(x, y)[0, 1] print(round(tss, 4), round(1 - (e @ e) / tss, 4), round(r, 4), round(r**2, 4)) rng = np.random.default_rng(353) x = np.arange(1, 21, dtype=float) y = 2 + 0.5 * x + rng.normal(0, 1, 20) print(round(np.corrcoef(x, y)[0, 1] ** 2, 4)) rng = np.random.default_rng(5108) x = rng.standard_normal(2000) y = 0.1 * x + rng.standard_normal(2000) print(round(np.cov(x, y)[0, 1] / np.var(x, ddof=1), 4), round(np.corrcoef(x, y)[0, 1] ** 2, 4))

⌘/Ctrl + Enter runs the cell

How far the estimates can stray

Lesson 3.5.3's fit, $y = 2.4162 + 0.4571x$ , missed the true line $y = 2 + 0.5x$ . How far off a fit like this can be is a question of statistics. The estimates are random variables (lesson 5.8.1): new errors would give new estimates.

Run the experiment again: keep the same 20 values of $x$ , draw new errors, and fit again, 10,000 times. In one seeded run the slopes average 0.5001 with standard deviation 0.0386; the intercepts average 1.9998 with standard deviation 0.4637. The middle 95% of the slopes runs from 0.4243 to 0.5758.

So the estimates are centred on the truth. The 0.4571 is an ordinary draw: 27.26% of the samples miss 0.5 by at least as much. The spread of the slope has a formula.

Rule: Spread of the estimated slope

\operatorname{sd}(\hat{\beta}_1) = \frac{\sigma}{\sqrt{\sum_i (x_i - \bar{x})^2}}

Under the model, $\hat{\beta}_1$ has mean $\beta_1$ and this standard deviation. It shrinks with less noise (smaller $\sigma$ ), more points, or $x$ values spread more widely. Lesson 5.10.3 replaces the unknown $\sigma$ by $s$ , which gives a standard error from one sample, and tests the slope.

Here $\sum_i (x_i - \bar{x})^2 = 665$ and $\sigma = 1$ , so the formula gives $\frac{1}{\sqrt{665}} = 0.0388$ , as the simulation found. Taking each $x$ four times halves it to 0.0194; so does spreading the $x$ values twice as wide. Lesson 5.8.2's $\frac{\sigma}{\sqrt{n}}$ for a mean behaved the same way.

0.5 - 0.4571

import numpy as np x = np.arange(1, 21, dtype=float) xc = x - x.mean() Y = 2 + 0.5 * x + np.random.default_rng(5109).normal(0, 1, (10_000, 20)) b1 = (Y - Y.mean(axis=1, keepdims=True)) @ xc / (xc @ xc) b0 = Y.mean(axis=1) - b1 * x.mean() print(round(b1.mean(), 4), round(b1.std(ddof=1), 4), round(b0.mean(), 4), round(b0.std(ddof=1), 4)) print(np.round(np.percentile(b1, [2.5, 97.5]), 4), round(np.mean(np.abs(b1 - 0.5) >= 0.5 - 0.457089), 4)) x4 = np.repeat(x, 4) print(xc @ xc, round(1 / np.sqrt(xc @ xc), 4), round(1 / np.sqrt(((x4 - x4.mean()) ** 2).sum()), 4))

⌘/Ctrl + Enter runs the cell

The explorer draws jointly normal pairs with correlation 0.6 and $\sigma_Y = 1$ , so the dashed line of conditional means has slope 0.6. The solid line is the least-squares fit to the dots, with their residuals.

Cut the draws to 20 and press "Draw again" a few times: the fitted slope wanders, as $\hat{\beta}_1$ did above. At 500 draws it stays close to the dashed line; then move $\rho$ and watch $r^2$ .

Explore

Each dot is one draw of the pair (X, Y). Move ρ and watch the cloud stretch along a line. The dashed line is E[Y | X = x], the average of Y for each x; turn on the fitted line to see least squares estimate it from the dots.

Correlation ρ = 0.6

Standard deviation of Y = 1

Number of draws = 200

Show the least-squares line and residuals

Correlation: true ρ = 0.6; sample r = 0.62
Slope of E[Y | X]: ρσ_Y/σ_X = 0.6 × 1 / 1 = 0.6
Fitted line: y = 0.011 + 0.61x, sum of squared residuals 136.18
Explained: r² = 0.385: the share of the variance of Y that the line accounts for

Regressing x on y is a different line

The regression of $y$ on $x$ predicts $y$ from $x$ . To predict $x$ from $y$ , swap their roles: the slope becomes $\frac{s_{xy}}{s_y^2} = r\frac{s_x}{s_y}$ , not $\frac{1}{\hat{\beta}_1}$ .

The product of the two slopes is $r^2$ . For the four points, $x$ on $y$ has slope 0.5132, while $\frac{1}{1.94} = 0.5155$ ; their product with 1.94 is 0.9957, which is $R^2$ . Both lines pass through $(\bar{x}, \bar{y})$ , and they coincide only when $r = \pm 1$ .

The weaker the link, the further apart the two lines. In case study 5.5.5, C on B has slope 0.6944 and B on C has 0.36, not $\frac{1}{0.6944} = 1.44$ : their product is $\rho^2 = 0.25$ .

\frac{1}{\hat{\beta}_1}

import numpy as np x = np.array([0.0, 1.0, 2.0, 3.0]) y = np.array([1.1, 2.9, 5.2, 6.8]) s_xy = np.cov(x, y)[0, 1] y_on_x, x_on_y = s_xy / np.var(x, ddof=1), s_xy / np.var(y, ddof=1) print(round(y_on_x, 4), round(x_on_y, 4), round(1 / y_on_x, 4), round(y_on_x * x_on_y, 4)) rho, sB, sC = 0.5, 0.18, 0.25 print(round(rho * sC / sB, 4), round(rho * sB / sC, 4), round(sB / (rho * sC), 4))

⌘/Ctrl + Enter runs the cell

Case study 5.10.5 estimates a stock's beta as the slope of such a line.

Common mistakes

Residuals are not errors. Residuals are gaps to the fitted line and add up to 0 exactly. Errors are gaps to the true line, which is unknown.
Divide RSS by $n - 2$ for $s$ , not by $n$ or $n - 1$ : the line used two estimates.
Regress the variable you want to predict on the one you know. The slope of $x$ on $y$ is not $\frac{1}{\hat{\beta}_1}$ unless $r = \pm 1$ .
A small $R^2$ is not a zero slope, and a large $R^2$ does not prove that a straight line is the right shape. $R^2$ measures how tightly the points hug the line.
The slope has units. With $x$ in percent rather than decimals, the slope is 100 times smaller; $R^2$ does not change.
An estimate is not the truth. One sample gave 0.4571 for a true 0.5. The slope's standard deviation, 0.0388, says how far such misses typically go.
Do not extrapolate far. The line describes the mean of $y$ over the range of $x$ in the data. Outside it, nothing in the data checks the straight line.

Simple linear regression

0 of 12 correct

1.
A sample has $s_{xy} = 6$ and $s_x^2 = 4$ . Find the least-squares slope $\hat{\beta}_1$
2.
The slope is $\hat{\beta}_1 = 1.5$ , and the means are $\bar{x} = 2$ and $\bar{y} = 7$ . Find $\hat{\beta}_0$
3.
Fit a line to $(1, 2)$ , $(2, 3)$ and $(3, 7)$ . Find $\hat{\beta}_1$
4.
For the same three points, find the residual at $x = 2$
5.
A sample has $r = 0.8$ , $s_x = 0.05$ and $s_y = 0.1$ . Find $\hat{\beta}_1$
6.
A regression has $\operatorname{TSS} = 50$ and $\operatorname{RSS} = 12$ . Find $R^2$
7.
With one regressor the sample correlation is $r = -0.5$ . Find $R^2$
8.
A line fitted to 12 points leaves $\operatorname{RSS} = 40$ . Find the residual standard deviation $s$
9.
The errors have $\sigma = 2$ and $\sum_i (x_i - \bar{x})^2 = 100$ . Find the standard deviation of $\hat{\beta}_1$
10.
Each $x$ value of that sample is now used four times. What is the standard deviation of $\hat{\beta}_1$ ?
11.
The regression of $y$ on $x$ has slope 2 and $R^2 = 0.5$ . Find the slope of the regression of $x$ on $y$
12.
With $x$ in decimals the slope is 0.5. What is it with $x$ in percent, so that 0.03 becomes 3?

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

y_i = \beta_0 + \beta_1x_i + \varepsilon_i

Simple linear regression

The model

Rule: Simple linear regression

The least-squares estimates

Rule: Least-squares estimates

A worked example

Every sign, and where it breaks

Fitted values and residuals

Errors are not residuals

Rule: Residual standard deviation

R²: how much the line explains

Rule: R squared

How far the estimates can stray

Rule: Spread of the estimated slope

Regressing x on y is a different line

Common mistakes

Simple linear regression

Summary card

Free further reading