5.10.4 Practice set: linear regression

Simple linear regression

Many questions use these five points:

$x_i$	1	2	3	4	5
$y_i$	2	4	5	4	5

Simple linear regression (lesson 5.10.1)

0 of 8 correct

1.
For the five points, find the least-squares slope $\hat{\beta}_1$
2.
For the five points, find the intercept $\hat{\beta}_0$
3.
For the five points, find the residual at $x = 3$
4.
The residuals of the five points are $-0.8, 0.6, 1, -0.6, -0.2$ and $\operatorname{TSS} = 6$ . Find $R^2$
5.
For the five points, $\operatorname{RSS} = 2.4$ . Find the residual standard deviation $s$ (type sqrt, or a decimal)
6.
For the five points, find the sample correlation $r$ (type sqrt, or a decimal)
7.
For the five points, find the slope of the regression of $x$ on $y$
8.
Errors have $\sigma = 3$ , and the $x$ values have $\sum_i (x_i - \bar{x})^2 = 36$ . Find the standard deviation of $\hat{\beta}_1$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

Regression in matrix form

Regression in matrix form (lesson 5.10.2)

0 of 7 correct

1.
For the five points with $X = [\mathbf{1}\ \mathbf{x}]$ , find the entry in row 2, column 2 of $X^\top X$
2.
For the same $X$ , find the second entry of $X^\top\mathbf{y}$
3.
A fit has $\hat{\boldsymbol{\beta}} = (1, 0.5, 2)$ for the columns $\mathbf{1}$ , $x_1$ and a dummy $x_2$ . Find the fitted value at $x_1 = 4$ in the group with $x_2 = 1$
4.
For the five points, find the leverage of the point at $x = 1$
5.
Monthly data has a column of ones and a dummy for each month but one. How many dummy columns are there?
6.
With both regressors, $\hat{\beta}_1 = 0.8$ and $\hat{\beta}_2 = -2$ . The slope of $x_2$ on $x_1$ is 0.3. Find the slope of $y$ on $x_1$ alone
7.
A regression with an intercept and 4 regressors fits 40 points. Find the sum of the leverages

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

Assumptions, t tests and F tests

Assumptions, t tests and F tests (lesson 5.10.3)

0 of 8 correct

1.
For the five points, $\operatorname{RSS} = 2.4$ with $n = 5$ and $k = 2$ . Find $s^2$
2.
For the five points, $[(X^\top X)^{-1}]_{22} = \frac{1}{10}$ . Find $\operatorname{se}(\hat{\beta}_1)$ (type sqrt, or a decimal)
3.
For the five points, find the t statistic for a slope of 0 (type sqrt, or a decimal)
4.
For the five points, find the F statistic that the slope is 0
5.
For the five points, $[(X^\top X)^{-1}]_{11} = \frac{1}{n} + \frac{\bar{x}^2}{\sum_i (x_i - \bar{x})^2}$ . Find the estimated variance of $\hat{\beta}_0$
6.
An estimate is 1.5 with standard error 0.25, and the t quantile is 3.2. Find the lower end of the 95% interval
7.
For the five points $R^2 = 0.6$ , $t = 2.1213$ , and with 3 degrees of freedom $t_{0.975} = 3.1824$ . Is the slope significant at 5%? Type 1 for yes, 0 for no
8.
The errors' variance grows with $x$ . Is $\hat{\boldsymbol{\beta}}$ still unbiased? Type 1 for yes, 0 for no

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

Coding exercise

Exercise

Least squares with standard errors and the F test

Write two functions with NumPy and scipy.stats, as in lessons 5.10.2 and 5.10.3. X is an $n \times k$ design matrix that includes a column of ones, in any position, and y holds the $n$ responses.

ols(X, y) returns (beta, se, r2): the least-squares coefficients in the order of the columns, their standard errors $s\sqrt{[(X^\top X)^{-1}]_{jj}}$ with $s^2 = \frac{\operatorname{RSS}}{n - k}$ , and $R^2$ .
f_test(X, y) returns (F, p): the F statistic that every coefficient except the intercept is 0, and its p-value. Use stats.f.sf with $k - 1$ and $n - k$ degrees of freedom.

Both raise ValueError when X is not 2-D, y is not 1-D, or their lengths differ. They also raise it when $n \le k$ , when no column of X is all ones, or when the columns are dependent.

The tests use the five points above, lesson 5.10.2's eight points and lesson 3.5.3's 20 noisy points. They also try the column of ones in last place, and each bad input.

See the tests

import numpy as np
from scipy import stats

TABLE_X = np.column_stack([np.ones(5), [1, 2, 3, 4, 5.0]])
TABLE_Y = np.array([2, 4, 5, 4, 5.0])

X8 = np.column_stack([np.ones(8), [1, 2, 3, 4, 3, 4, 5, 6.0], [0, 0, 0, 0, 1, 1, 1, 1.0]])
Y8 = np.array([3.5, 6, 5.5, 7, 1.5, 1.5, 2.5, 4.5])


def _noisy_line():
    rng = np.random.default_rng(353)
    x = np.arange(1, 21, dtype=float)
    return np.column_stack([np.ones(20), x]), 2 + 0.5 * x + rng.normal(0, 1, 20)


def _raises(f, *args, what=""):
    try:
        f(*args)
    except ValueError:
        return
    raise AssertionError(f"expected ValueError: {what}")


def test_table():
    """The practice set's five points give the line 2.2 + 0.6x, standard errors √0.88 and √0.08, and R² = 0.6"""
    beta, se, r2 = ols(TABLE_X, TABLE_Y)
    expect_close(beta, [2.2, 0.6], what="beta")
    expect_close(se, [np.sqrt(0.88), np.sqrt(0.08)], what="se")
    expect_close(r2, 0.6, what="r2")


def test_table_f():
    """With one slope, F is t², 4.5, with p-value 0.124 on 1 and 3 degrees of freedom"""
    F, p = f_test(TABLE_X, TABLE_Y)
    expect_close(F, 4.5, what="F")
    expect_close(p, stats.f.sf(4.5, 1, 3), what="p")


def test_two_regressors():
    """Lesson 5.10.2's eight points give (3, 1, −5) with standard errors 0.6614, 0.2236 and 0.6708"""
    beta, se, r2 = ols(X8, Y8)
    expect_close(beta, [3, 1, -5], what="beta")
    expect_close(se, np.sqrt(0.5 * np.array([7 / 8, 1 / 10, 9 / 10])), what="se")
    expect_close(r2, 1 - 2.5 / 30.5, what="r2")


def test_two_regressors_f():
    """The eight points give F = 28 on 2 and 5 degrees of freedom, p about 0.0019"""
    F, p = f_test(X8, Y8)
    expect_close(F, 28.0, what="F")
    expect_close(p, stats.f.sf(28.0, 2, 5), what="p")


def test_noisy_line():
    """Lesson 3.5.3's 20 points give slope 0.4571 with standard error 0.0449"""
    X, y = _noisy_line()
    beta, se, r2 = ols(X, y)
    expect_close(beta, np.linalg.lstsq(X, y, rcond=None)[0], what="beta")
    expect_close(se[1], 0.044869, rel=1e-4, what="se of the slope")
    expect_close(r2, np.corrcoef(X[:, 1], y)[0, 1] ** 2, what="r2")


def test_ones_anywhere():
    """The column of ones may be any column: putting it last reverses the order of beta"""
    beta, se, r2 = ols(TABLE_X[:, ::-1], TABLE_Y)
    expect_close(beta, [0.6, 2.2], what="beta with the ones last")
    expect_close(r2, 0.6, what="r2")


def test_bad_shapes():
    """Rows that do not match y, or as many coefficients as observations, raise ValueError"""
    _raises(ols, TABLE_X, TABLE_Y[:4], what="5 rows and 4 responses")
    _raises(ols, TABLE_X[:2], TABLE_Y[:2], what="2 observations and 2 coefficients")
    _raises(f_test, TABLE_X, TABLE_Y[:4], what="f_test with 5 rows and 4 responses")


def test_dependent_columns():
    """A dummy for every group beside the column of ones raises ValueError"""
    trap = np.column_stack([np.ones(8), X8[:, 2], 1 - X8[:, 2]])
    _raises(ols, trap, Y8, what="the dummy trap")
    _raises(f_test, trap, Y8, what="f_test on the dummy trap")


def test_no_intercept():
    """A design matrix without a column of ones raises ValueError"""
    _raises(ols, X8[:, 1:], Y8, what="no column of ones")

Simple linear regression

Regression in matrix form

Assumptions, t tests and F tests

Coding exercise

Free further reading