5.10.3 Assumptions, t tests and F tests

In this lesson

The usual assumptions of linear regression in plain words, and what each one buys
Why $\hat{\boldsymbol{\beta}}$ is centred on $\boldsymbol{\beta}$ , and its covariance matrix $\sigma^2(X^\top X)^{-1}$
The estimate $s^2 = \operatorname{RSS}/(n - k)$ , standard errors, and the t statistic with $n - k$ degrees of freedom
Confidence intervals and t tests for one coefficient
The F test that every slope is 0, and when it disagrees with the t tests
What breaks when the errors change in size or are correlated with each other
Checking every step with NumPy and scipy.stats

You need regression in matrix form from lesson 5.10.2 and the rule $\operatorname{Cov}(A\mathbf{X}) = A\Sigma A^\top$ from lesson 5.5.3. Intervals and tests come from lessons 5.8.2 and 5.8.3, and the t and $\chi^2$ distributions from lesson 5.4.4.

The assumptions

Lesson 5.10.2 fitted $\mathbf{y} = X\boldsymbol{\beta} + \boldsymbol{\varepsilon}$ by least squares, which needs no assumptions at all. To say how far $\hat{\boldsymbol{\beta}}$ may be from $\boldsymbol{\beta}$ , you need a model of the errors. These are the usual assumptions, with the $x$ values treated as fixed numbers, as lesson 5.10.1's simulation kept the same 20 values of $x$ .

Linear mean. $\mathbb{E}[\mathbf{y}] = X\boldsymbol{\beta}$ : the columns of $X$ capture how the mean of $y$ depends on the regressors.
Independent columns. $X$ has rank $k$ , so $(X^\top X)^{-1}$ exists (lesson 5.10.2).
Errors unrelated to the regressors. Each $\varepsilon_i$ has mean 0, whatever the regressors are. This fails when a regressor that matters is left out and moves with one that is in: lesson 5.10.2's omitted-variable bias.
Constant variance. $\operatorname{Var}(\varepsilon_i) = \sigma^2$ , the same for every observation.
Uncorrelated errors. $\operatorname{Cov}(\varepsilon_i, \varepsilon_j) = 0$ for $i \neq j$ : one error says nothing about another.
Normal errors. Each $\varepsilon_i$ is normal. This one is needed only for the exact t and F distributions below.

Rule: What the assumptions buy

\begin{gathered} \mathbb{E}[\hat{\boldsymbol{\beta}}] = \boldsymbol{\beta} \\[4pt] \operatorname{Cov}(\hat{\boldsymbol{\beta}}) = \sigma^2(X^\top X)^{-1} \end{gathered}

Assumptions 1 to 3 make $\hat{\boldsymbol{\beta}}$ unbiased (lesson 5.8.1): centred on the truth. Adding 4 and 5 gives its covariance matrix, whose diagonal holds the variances of the coefficients. Adding 6 makes $\hat{\boldsymbol{\beta}}$ jointly normal.

The covariance of the estimates

Three steps give the rule, each using a lesson you have met. Write $A = (X^\top X)^{-1}X^\top$ , a fixed $k \times n$ matrix, so that $\hat{\boldsymbol{\beta}} = A\mathbf{y}$ .

1
$\hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + A\boldsymbol{\varepsilon}$
2
$\mathbb{E}[\hat{\boldsymbol{\beta}}] = \boldsymbol{\beta} + A\,\mathbb{E}[\boldsymbol{\varepsilon}] = \boldsymbol{\beta}$
3
$\operatorname{Cov}(\hat{\boldsymbol{\beta}}) = \sigma^2AA^\top = \sigma^2(X^\top X)^{-1}$

For lesson 3.5.3's 20 values of $x$ , with $\sigma = 1$ :

\sigma^2(X^\top X)^{-1} = \begin{pmatrix} 0.2158 & -0.0158 \\ -0.0158 & 0.0015 \end{pmatrix}

The square roots of the diagonal are 0.4645 for the intercept and 0.0388 for the slope: lesson 5.10.1's $\frac{\sigma}{\sqrt{665}}$ . Lesson 5.10.1's 10,000 simulated samples give the sample covariance matrix $\begin{pmatrix} 0.215 & -0.0157 \\ -0.0157 & 0.0015 \end{pmatrix}$ , close to it.

The off-diagonal entry is negative: the correlation of $\hat{\beta}_0$ and $\hat{\beta}_1$ is $-0.8765$ . Every fitted line passes through the mean point $(\bar{x}, \bar{y})$ , here with $\bar{x} = 10.5$ . A line that comes out steeper must cross $x = 0$ lower, so a high slope comes with a low intercept.

(X^\top X)^{-1}

import numpy as np x = np.arange(1, 21, dtype=float) X = np.column_stack([np.ones(20), x]) V = np.linalg.inv(X.T @ X) print(np.round(V, 4), np.round(np.sqrt(np.diag(V)), 4), round(V[0, 1] / np.sqrt(V[0, 0] * V[1, 1]), 4)) xc = x - x.mean() Y = 2 + 0.5 * x + np.random.default_rng(5109).normal(0, 1, (10_000, 20)) b1 = (Y - Y.mean(axis=1, keepdims=True)) @ xc / (xc @ xc) b0 = Y.mean(axis=1) - b1 * x.mean() print(np.round(np.cov(b0, b1), 4), round(np.corrcoef(b0, b1)[0, 1], 4))

⌘/Ctrl + Enter runs the cell

Standard errors and t statistics

The rule needs $\sigma^2$ , which is unknown. Estimate it from the residuals, as lesson 5.10.1 did with $n - 2$ ; with $k$ coefficients, divide by $n - k$ . The square root of each diagonal entry then becomes a standard error.

Maximum likelihood divides RSS by $n$ instead (lesson 5.9.2), which makes the estimate too small on average. Dividing by $n - k$ removes that bias, as $n - 1$ does for a sample variance (lesson 5.8.1).

Rule: Standard errors and the t statistic

\begin{gathered} s^2 = \frac{\operatorname{RSS}}{n - k} \\[4pt] \operatorname{se}(\hat{\beta}_j) = s\sqrt{\big[(X^\top X)^{-1}\big]_{jj}} \\[4pt] t = \frac{\hat{\beta}_j - b}{\operatorname{se}(\hat{\beta}_j)} \end{gathered}

$[(X^\top X)^{-1}]_{jj}$ is the diagonal entry for coefficient $j$ , and $n - k$ is the number of degrees of freedom. Under all six assumptions, if the true $\beta_j$ is the number $b$ , then $t$ has the t distribution with $n - k$ degrees of freedom (lesson 5.4.4).

That gives the two tools of module 5.8, which read as follows for a coefficient. Lesson 5.8.2 built a 95% interval as the estimate plus or minus a t quantile times its standard error. Lesson 5.8.3 tested a value by its t statistic and p-value.

\begin{gathered} \hat{\beta}_j \pm t_{0.975,\,n-k}\operatorname{se}(\hat{\beta}_j) \\[4pt] p = 2\,P(T_{n-k} \ge \lvert t \rvert) \end{gathered}

Here $t_{0.975,\,n-k}$ is the 0.975-quantile of the t distribution, t.ppf(0.975, n - k), and $T_{n-k}$ a variable with that distribution.

How far the fit may be from the truth

Lesson 3.5.3's 20 points gave $\hat{\boldsymbol{\beta}} = (2.4162, 0.4571)$ . With $n - k = 18$ degrees of freedom, $s = 1.1571$ , so the slope's standard error is $\frac{1.1571}{\sqrt{665}} = 0.0449$ and the intercept's is 0.5375. The quantile is $t_{0.975,\,18} = 2.1009$ .

	estimate	standard error	95% interval	t for the truth	p
intercept	2.4162	0.5375	1.2869 to 3.5454	0.7742	0.4488
slope	0.4571	0.0449	0.3628 to 0.5513	$-0.9571$	0.3512

Both intervals cover the true values, 2 and 0.5. The t statistics test those true values: a p-value of 0.3512 says a miss as large as $0.5 - 0.4571$ is common. This is how far the fit may be from the truth, from one sample alone.

The test that matters most in practice is whether the slope is 0 at all. Here $t = \frac{0.4571}{0.0449} = 10.1862$ , and the p-value is about $7 \times 10^{-9}$ : $y$ does depend on $x$ .

s

import numpy as np from scipy import stats rng = np.random.default_rng(353) x = np.arange(1, 21, dtype=float) y = 2 + 0.5 * x + rng.normal(0, 1, 20) X = np.column_stack([np.ones(20), x]) n, k = X.shape b = np.linalg.lstsq(X, y, rcond=None)[0] e = y - X @ b s = np.sqrt(e @ e / (n - k)) se = s * np.sqrt(np.diag(np.linalg.inv(X.T @ X))) q = stats.t.ppf(0.975, n - k) print(np.round(b, 4), round(s, 4), np.round(se, 4), round(q, 4)) print(np.round(b - q * se, 4), np.round(b + q * se, 4)) t_true = (b - np.array([2, 0.5])) / se print(np.round(t_true, 4), np.round(2 * stats.t.sf(np.abs(t_true), n - k), 4)) print(np.round(b / se, 4), f"{2 * stats.t.sf(b[1] / se[1], n - k):.0e}", round((b[1] / se[1]) ** 2, 4))

⌘/Ctrl + Enter runs the cell

Why the t quantile, not 1.96

Lesson 5.10.1's 10,000 samples can check the interval. With each sample's own $s$ , the interval $\hat{\beta}_1 \pm 2.1009\operatorname{se}$ covers 0.5 in 95.01% of them. With 1.96 in place of 2.1009 it covers only 93.54%.

The standard error uses $s$ , which varies from sample to sample, and that widens the spread of $t$ : lesson 5.4.4's fatter tails. Across the samples, $s^2$ averages 0.9982, close to $\sigma^2 = 1$ , which is why $n - k$ is the divisor. With many degrees of freedom the t quantile comes close to 1.96.

s

import numpy as np from scipy import stats x = np.arange(1, 21, dtype=float) xc = x - x.mean() Y = 2 + 0.5 * x + np.random.default_rng(5109).normal(0, 1, (10_000, 20)) b1 = (Y - Y.mean(axis=1, keepdims=True)) @ xc / (xc @ xc) b0 = Y.mean(axis=1) - b1 * x.mean() resid = Y - b0[:, None] - b1[:, None] * x s2 = (resid**2).sum(axis=1) / 18 t = (b1 - 0.5) / np.sqrt(s2 / (xc @ xc)) print(round(stats.t.ppf(0.975, 18), 4), round(np.mean(np.abs(t) <= stats.t.ppf(0.975, 18)), 4), round(np.mean(np.abs(t) <= 1.96), 4)) print(round(s2.mean(), 4))

⌘/Ctrl + Enter runs the cell

Two regressors

Lesson 5.10.2's eight points have $n - k = 5$ and $\operatorname{RSS} = 2.5$ , so $s^2 = 0.5$ . The diagonal of $(X^\top X)^{-1}$ is $\frac{7}{8}, \frac{1}{10}, \frac{9}{10}$ , so $\operatorname{se}(\hat{\beta}_1) = \sqrt{0.5 \times 0.1} = 0.2236$ .

	estimate	standard error	t for 0	p
intercept	3	0.6614	4.5356	0.0062
$x_1$	1	0.2236	4.4721	0.0066
group $x_2$	$-5$	0.6708	$-7.4536$	0.0007

With 5 degrees of freedom the quantile is 2.5706, so the 95% interval for $\beta_1$ runs from 0.4252 to 1.5748.

The F test

A t test asks about one coefficient. The F test asks whether any regressor matters: its null hypothesis is that every slope is 0, $\beta_1 = \cdots = \beta_p = 0$ , so that only the intercept is left.

Under that null the best fit is the flat line at $\bar{y}$ , which leaves TSS. The full model leaves RSS. The F statistic compares the drop, per slope, with what is left, per degree of freedom.

Rule: The F test that all slopes are 0

\begin{gathered} F = \frac{(\operatorname{TSS} - \operatorname{RSS})/(k - 1)}{\operatorname{RSS}/(n - k)} \\[6pt] = \frac{R^2/(k - 1)}{(1 - R^2)/(n - k)} \end{gathered}

Under the null and the six assumptions, $F$ has the F distribution with $k - 1$ and $n - k$ degrees of freedom. That is a ratio of two independent $\chi^2$ variables (lesson 5.4.4), each divided by its degrees of freedom. Large $F$ is evidence against the null; the p-value is f.sf(F, k - 1, n - k).

For the eight points, $\operatorname{TSS} = 30.5$ and $\operatorname{RSS} = 2.5$ :

F = \frac{28/2}{2.5/5} = 28

With 2 and 5 degrees of freedom, the p-value is 0.0019; the 5% critical value is 5.7861. With a single slope, $F = t^2$ : lesson 3.5.3's points give $F = 103.758 = 10.1862^2$ , with the same p-value as the t test.

When t and F disagree

Take 30 invented points with two regressors that are almost the same: $x_2$ is $x_1$ plus a little noise, and their correlation is 0.9993. The true model is $y = 1 + 0.5x_1 + 0.5x_2 + \varepsilon$ .

The fit gives $\hat{\beta}_1 = 3.3173$ and $\hat{\beta}_2 = -2.425$ , far from the truth, with standard errors 3.5078 and 3.4812. Neither t test finds anything: their p-values are 0.353 and 0.492. Yet $F = 21.1101$ , with p-value $3 \times 10^{-6}$ : together the two regressors matter a great deal.

The data cannot tell $x_1$ from $x_2$ , so it cannot split their effect, but it pins down the total. The sum $\hat{\beta}_1 + \hat{\beta}_2 = 0.8924$ has standard error 0.1376, close to the true 1.

This is collinearity, and large standard errors beside a significant F are its mark. Keep one of the two regressors, or test their combined effect.

(X^\top X)^{-1}

import numpy as np from scipy import stats def ols(X, y): n, k = X.shape b = np.linalg.lstsq(X, y, rcond=None)[0] e = y - X @ b V = (e @ e / (n - k)) * np.linalg.inv(X.T @ X) r2 = 1 - (e @ e) / ((y - y.mean()) ** 2).sum() F = (r2 / (k - 1)) / ((1 - r2) / (n - k)) p_t = 2 * stats.t.sf(np.abs(b / np.sqrt(np.diag(V))), n - k) return b, V, p_t, F, stats.f.sf(F, k - 1, n - k) x1 = np.array([1, 2, 3, 4, 3, 4, 5, 6.0]) x2 = np.array([0, 0, 0, 0, 1, 1, 1, 1.0]) y = np.array([3.5, 6, 5.5, 7, 1.5, 1.5, 2.5, 4.5]) X = np.column_stack([np.ones(8), x1, x2]) b, V, p_t, F, p_F = ols(X, y) se = np.sqrt(np.diag(V)) print(np.round(np.diag(np.linalg.inv(X.T @ X)), 4), np.round(se, 4), np.round(b / se, 4), np.round(p_t, 4)) q = stats.t.ppf(0.975, 5) print(round(q, 4), round(b[1] - q * se[1], 4), round(b[1] + q * se[1], 4)) print(round(F, 4), round(p_F, 4), round(stats.f.ppf(0.95, 2, 5), 4)) g = np.random.default_rng(5138) u = g.standard_normal(30) w = u + 0.05 * g.standard_normal(30) v = 1 + 0.5 * u + 0.5 * w + g.standard_normal(30) b, V, p_t, F, p_F = ols(np.column_stack([np.ones(30), u, w]), v) a = np.array([0, 1, 1.0]) print(round(np.corrcoef(u, w)[0, 1], 4), np.round(b, 4), np.round(np.sqrt(np.diag(V)), 4), np.round(p_t, 3)) print(round(F, 4), f"{p_F:.0e}", round(a @ b, 4), round(np.sqrt(a @ V @ a), 4))

⌘/Ctrl + Enter runs the cell

When the assumptions fail

Least squares still fits a line when the assumptions fail. What goes wrong is the statistics: the standard errors, and so every interval and p-value. Plot the residuals to see which assumption is in doubt.

Residuals from three invented samples: in the first the assumptions hold, in the second the spread grows with $x$ . In the third each error is close to its neighbour, so the residuals run in long stretches above or below 0.

Changing variance. Suppose the errors' standard deviation grows in proportion to $x$ , scaled so that their average variance is still 1. Lesson 5.10.1's experiment then gives slopes whose true spread is 0.0431, but the usual standard error averages 0.0379. The nominal 95% intervals cover 0.5 only 92.26% of the time.

Correlated errors. Data in time order often has errors close to their neighbours: make each error correlated 0.5 with the one before (a lag-one correlation). The slopes' true spread becomes 0.0606, while the usual standard error averages 0.0344. The intervals cover only 75.28% of the time, and 50.93% at a lag-one correlation of 0.8.

With the errors as assumed, the same experiment gives a true spread of 0.0391, an average standard error of 0.0381 and coverage of 94.92%. In both failures $\hat{\boldsymbol{\beta}}$ is still unbiased, but $\sigma^2(X^\top X)^{-1}$ is the wrong covariance matrix. Here it was too small, so t statistics came out too large and slopes looked significant too often.

Rule: What each failure does

\begin{gathered} \text{changing variance or} \\ \text{correlated errors:} \\ \hat{\boldsymbol{\beta}} \text{ unbiased,} \\ \text{standard errors wrong} \\[6pt] \text{errors related to a regressor:} \\ \hat{\boldsymbol{\beta}} \text{ biased} \end{gathered}

Changing variance is called heteroscedasticity, and correlated errors autocorrelation. Robust standard errors (White's for changing variance, Newey and West's for correlation too) estimate the spread without those assumptions. No standard error repairs bias from a left-out regressor.

Non-normal errors matter less. With many observations the central limit theorem (lesson 5.7.3) makes $\hat{\boldsymbol{\beta}}$ close to normal anyway, so t and F are close to right. Fat tails and outliers are the exception: a single wild point with high leverage (lesson 5.10.2) can move the fit.

x

import numpy as np from scipy import stats x = np.arange(1, 21, dtype=float) xc = x - x.mean() q = stats.t.ppf(0.975, 18) def coverage(E): Y = 2 + 0.5 * x + E b1 = (Y - Y.mean(axis=1, keepdims=True)) @ xc / (xc @ xc) b0 = Y.mean(axis=1) - b1 * x.mean() resid = Y - b0[:, None] - b1[:, None] * x se = np.sqrt((resid**2).sum(axis=1) / 18 / (xc @ xc)) return round(b1.std(ddof=1), 4), round(se.mean(), 4), round(np.mean(np.abs(b1 - 0.5) <= q * se), 4) print(*coverage(np.random.default_rng(5132).normal(0, 1, (10_000, 20)))) print(*coverage(np.random.default_rng(5131).normal(0, 1, (10_000, 20)) * x / np.sqrt((x**2).mean()))) for rho in (0.5, 0.8): Z = np.random.default_rng(5133).normal(0, 1, (10_000, 20)) E = Z.copy() for t in range(1, 20): E[:, t] = rho * E[:, t - 1] + np.sqrt(1 - rho**2) * Z[:, t] print(rho, *coverage(E))

⌘/Ctrl + Enter runs the cell

Case study 5.10.5 applies these tests to a stock's beta and alpha.

Common mistakes

Divide RSS by $n - k$ , the number of observations minus the number of coefficients, intercept included.
Use the t quantile with $n - k$ degrees of freedom, not 1.96, for a small sample: with 18 degrees of freedom, 1.96 covers 93.54%, not 95%.
A large p-value does not prove the null. Lesson 3.5.3's test of a slope of 0.5 gave $p = 0.3512$ . Every value inside the interval, 0.3628 to 0.5513, would also pass.
Insignificant t statistics do not mean no regressor matters. Collinear regressors can each look useless while the F test is overwhelming.
Standard errors rest on assumptions 4 and 5. With changing variance or correlated errors they can be far too small; look at the residuals.
No standard error fixes a left-out regressor. That makes $\hat{\boldsymbol{\beta}}$ itself biased (lesson 5.10.2).

Assumptions, t tests and F tests

0 of 12 correct

1.
A regression with $n = 22$ and $k = 3$ leaves $\operatorname{RSS} = 38$ . Find $s^2$
2.
$s^2 = 0.5$ and the diagonal entry of $(X^\top X)^{-1}$ for $\hat{\beta}_1$ is 0.1. Find $\operatorname{se}(\hat{\beta}_1)$ (type sqrt, or a decimal)
3.
$\sigma^2 = 4$ and the diagonal entry of $(X^\top X)^{-1}$ for $\hat{\beta}_1$ is 0.25. Find the standard deviation of $\hat{\beta}_1$
4.
A coefficient is estimated at 1.2 with standard error 0.4. Find the t statistic for the null that it is 0
5.
For the same estimate, find the t statistic for the null that the coefficient is 1
6.
A regression has 60 observations, an intercept and one regressor. How many degrees of freedom does its t statistic have?
7.
An estimate is 0.5 with standard error 0.1, and the t quantile is 2. Find the upper end of the 95% interval
8.
A regression has $n = 23$ , $k = 3$ and $R^2 = 0.5$ . Find the F statistic that all slopes are 0
9.
A regression with one slope has $t = 3$ for that slope. Find its F statistic
10.
The usual standard error of a slope is 0.03, but correlated errors make its true spread 0.06. A reported t of 4 is really about how many true standard deviations?
11.
Twenty independent 95% intervals are built, each with all assumptions holding. How many do you expect to miss the truth?
12.
Two nearly identical regressors have t statistics 0.9 and $-0.7$ but $F = 21$ with a tiny p-value. Do the regressors jointly matter? Type 1 for yes, 0 for no

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\hat{\boldsymbol{\beta}} = \boldsymbol{\beta} + (X^\top X)^{-1}X^\top\boldsymbol{\varepsilon}

Assumptions, t tests and F tests

The assumptions

Rule: What the assumptions buy

The covariance of the estimates

Standard errors and t statistics

Rule: Standard errors and the t statistic

How far the fit may be from the truth

Why the t quantile, not 1.96

Two regressors

The F test

Rule: The F test that all slopes are 0

When t and F disagree

When the assumptions fail

Rule: What each failure does

Common mistakes

Assumptions, t tests and F tests

Summary card

Free further reading