3.5.2 Orthogonal projection

The shadow as a vector

Lesson 3.1.2 shone a light straight down onto the line through $\mathbf{u}$ . The shadow of $\mathbf{v}$ had signed length $\frac{\mathbf{u} \cdot \mathbf{v}}{\lVert \mathbf{u} \rVert}$ . The projection of $\mathbf{v}$ onto the line is that shadow as a vector: its signed length times the unit vector $\frac{\mathbf{u}}{\lVert \mathbf{u} \rVert}$ .

That divides by $\lVert \mathbf{u} \rVert$ twice, which is dividing by $\lVert \mathbf{u} \rVert^2 = \mathbf{u} \cdot \mathbf{u}$ . The formula needs no square roots.

Rule: Projection onto a line

\mathbf{p} = \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{u} \cdot \mathbf{u}}\,\mathbf{u}

$\mathbf{p}$ is a multiple of $\mathbf{u}$ , so it lies on the line through $\mathbf{u}$ . The number in front is the coefficient of the projection. It needs $\mathbf{u} \neq \mathbf{0}$ : the zero vector gives no line.

For $\mathbf{u} = (3, 1)$ and $\mathbf{v} = (1, 2)$ : $\mathbf{u} \cdot \mathbf{v} = 5$ and $\mathbf{u} \cdot \mathbf{u} = 10$ , so $\mathbf{p} = 0.5\,\mathbf{u} = (1.5, 0.5)$ . Its length is $\sqrt{2.5} \approx 1.5811$ , the shadow length of lesson 3.1.2.

The cases follow from the sign and size of the coefficient:

Behind the origin. $(-2, 1)$ has $\mathbf{u} \cdot \mathbf{v} = -5$ , so $\mathbf{p} = -0.5\,\mathbf{u} = (-1.5, -0.5)$ : the coefficient is negative.
Already on the line. $(6, 2) = 2\mathbf{u}$ projects onto itself.
Orthogonal to the line. $(-1, 3)$ has $\mathbf{u} \cdot \mathbf{v} = 0$ , so $\mathbf{p} = \mathbf{0}$ .
Only the line matters. With $2\mathbf{u}$ in place of $\mathbf{u}$ the coefficient halves to $\frac{10}{40} = 0.25$ , and $0.25 \times 2\mathbf{u}$ is the same $\mathbf{p}$ . So is $-0.5 \times (-\mathbf{u})$ .

The residual is orthogonal

The residual is what is left of $\mathbf{v}$ after the projection: $\mathbf{r} = \mathbf{v} - \mathbf{p}$ . Here $\mathbf{r} = (1, 2) - (1.5, 0.5) = (-0.5, 1.5)$ , and $\mathbf{u} \cdot \mathbf{r} = -1.5 + 1.5 = 0$ . The residual is orthogonal to the line (lesson 3.1.2).

That is how the coefficient is chosen. Ask for the multiple $c\,\mathbf{u}$ that leaves a residual orthogonal to $\mathbf{u}$ :

\begin{gathered} \mathbf{u} \cdot (\mathbf{v} - c\,\mathbf{u}) = 0 \\[4pt] \mathbf{u} \cdot \mathbf{v} - c\,(\mathbf{u} \cdot \mathbf{u}) = 0 \\[4pt] c = \frac{\mathbf{u} \cdot \mathbf{v}}{\mathbf{u} \cdot \mathbf{u}} \end{gathered}

So $\mathbf{v} = \mathbf{p} + \mathbf{r}$ splits $\mathbf{v}$ into a part along $\mathbf{u}$ and a part at right angles to it. The two parts are the legs of a right-angled triangle with hypotenuse $\mathbf{v}$ , so Pythagoras gives $\lVert \mathbf{v} \rVert^2 = \lVert \mathbf{p} \rVert^2 + \lVert \mathbf{r} \rVert^2$ : here $5 = 2.5 + 2.5$ .

An example in space. Project $\mathbf{v} = (3, 3, 0)$ onto the line through $\mathbf{u} = (1, 2, 2)$ . The dot product is 9 and $\mathbf{u} \cdot \mathbf{u} = 9$ , so $\mathbf{p} = \mathbf{u} = (1, 2, 2)$ and $\mathbf{r} = (2, 1, -2)$ . Check: $\mathbf{u} \cdot \mathbf{r} = 2 + 2 - 4 = 0$ , and $18 = 9 + 9$ .

The nearest point on the line

Every point of the line is $t\,\mathbf{u}$ for some number $t$ . Which one is closest to $\mathbf{v} = (1, 2)$ ? Here are five:

$t$	0	0.25	0.5	0.75	1
point $t\,\mathbf{u}$	$(0, 0)$	$(0.75, 0.25)$	$(1.5, 0.5)$	$(2.25, 0.75)$	$(3, 1)$
distance to $\mathbf{v}$	2.2361	1.7678	1.5811	1.7678	2.2361

The smallest is at $t = 0.5$ , the projection. Algebra shows why: the squared distance is $\lVert \mathbf{v} - t\,\mathbf{u} \rVert^2 = 5 - 10t + 10t^2$ , a parabola. Its lowest point is at $t = \frac{10}{2 \times 10} = 0.5$ (lesson 1.2.2).

The projection $\mathbf{p}$ is the point of the line nearest $\mathbf{v}$ . The residual $\mathbf{r}$ meets the line at a right angle; any other point of the line is further from $\mathbf{v}$ .

The right angle is the reason in every case. For any other point $\mathbf{q}$ on the line, $\mathbf{v} - \mathbf{q} = \mathbf{r} + (\mathbf{p} - \mathbf{q})$ , and those two parts are orthogonal. So $\lVert \mathbf{v} - \mathbf{q} \rVert^2 = \lVert \mathbf{r} \rVert^2 + \lVert \mathbf{p} - \mathbf{q} \rVert^2$ , which is never less than $\lVert \mathbf{r} \rVert^2$ .

Rule: The nearest point

\begin{gathered} \mathbf{p} \text{ is the point of the line} \\ \text{nearest } \mathbf{v} \\[4pt] \text{distance} = \lVert \mathbf{v} - \mathbf{p} \rVert \end{gathered}

The distance from a point to a line is the length of the residual, measured at right angles to the line.

The explorer starts on this example, with the projection as the dark arrow. Drag $\mathbf{v}$ onto the line and the projection becomes $\mathbf{v}$ ; drag it to a right angle with $\mathbf{u}$ and the projection shrinks to $\mathbf{0}$ . Then stretch or reverse $\mathbf{u}$ : the projection does not move.

Explore

Drag the tips of u and v. Watch the sign of u · v as the angle passes 90°. The dark arrow is the projection of v on u: the shadow v casts on the line through u.

u₁ = 3

u₂ = 1

v₁ = 1

v₂ = 2

u: (3, 1), length 3.1623
v: (1, 2), length 2.2361
u · v: 3 × 1 + 1 × 2 = 5
Angle: cos θ = 0.7071, so θ = 45°
Projection of v on u: (1.5, 0.5), signed length u · v / |u| = 1.5811
So: The dot product is positive, so the angle is less than 90°.

\mathbf{p}

import numpy as np def proj(u, v): return (u @ v) / (u @ u) * u u = np.array([3.0, 1.0]) v = np.array([1.0, 2.0]) p = proj(u, v) r = v - p print(p, r, u @ r) print(v @ v, p @ p, r @ r) print(proj(u, np.array([-2.0, 1.0])), proj(2 * u, v), proj(-u, v)) for t in [0, 0.25, 0.5, 0.75, 1]: print(t, t * u, round(np.linalg.norm(v - t * u), 4)) u3 = np.array([1.0, 2.0, 2.0]) v3 = np.array([3.0, 3.0, 0.0]) print(proj(u3, v3), v3 - proj(u3, v3), u3 @ (v3 - proj(u3, v3)))

⌘/Ctrl + Enter runs the cell

Projection onto a plane

A plane through the origin is the span of two independent vectors (lesson 3.5.1). Put them in the columns of a matrix $A$ . Every point of the plane is $A\mathbf{x}$ for some $\mathbf{x}$ : the plane is the column space of $A$ .

To project $\mathbf{b}$ onto the plane, look for $\mathbf{p} = A\hat{\mathbf{x}}$ whose residual $\mathbf{b} - \mathbf{p}$ is orthogonal to the whole plane. It is enough to be orthogonal to both columns, since every vector of the plane is a combination of them. ( $\hat{\mathbf{x}}$ is read "x hat".)

The two conditions $\mathbf{a}_1 \cdot (\mathbf{b} - A\hat{\mathbf{x}}) = 0$ and $\mathbf{a}_2 \cdot (\mathbf{b} - A\hat{\mathbf{x}}) = 0$ are the two rows of one matrix equation. The rows of $A^\top$ are the columns of $A$ (lesson 3.2.3), so

\begin{gathered} A^\top(\mathbf{b} - A\hat{\mathbf{x}}) = \mathbf{0} \\[4pt] A^\top A\,\hat{\mathbf{x}} = A^\top\mathbf{b} \end{gathered}

$A^\top A$ is square, and it has an inverse when the columns of $A$ are independent. Lesson 3.5.3 meets the case where they are not.

Rule: Projection onto a column space

\begin{gathered} \hat{\mathbf{x}} = (A^\top A)^{-1}A^\top\mathbf{b} \\[4pt] \mathbf{p} = A\hat{\mathbf{x}} \\[4pt] A^\top(\mathbf{b} - \mathbf{p}) = \mathbf{0} \end{gathered}

$A$ has independent columns. $\mathbf{p}$ is the point of the column space nearest $\mathbf{b}$ , and the residual is orthogonal to every column. With one column $\mathbf{u}$ , this is the rule for a line: $A^\top A = \mathbf{u} \cdot \mathbf{u}$ and $A^\top\mathbf{b} = \mathbf{u} \cdot \mathbf{b}$ .

A worked example

Take lesson 3.5.1's plane, spanned by $\mathbf{a}_1 = (1, 0, 1)$ and $\mathbf{a}_2 = (0, 1, 1)$ , and the point $\mathbf{b} = (0, 1, 4)$ . It is off the plane: its third component is not the sum of the first two.

\begin{gathered} A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix} \\[6pt] A^\top A = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}, \quad A^\top\mathbf{b} = \begin{pmatrix} 4 \\ 5 \end{pmatrix} \end{gathered}

$A^\top A$ is the symmetric matrix $S$ of lesson 3.2.3, as every $A^\top A$ is symmetric. Solving $2x_1 + x_2 = 4$ and $x_1 + 2x_2 = 5$ gives $\hat{\mathbf{x}} = (1, 2)$ . So $\mathbf{p} = \mathbf{a}_1 + 2\mathbf{a}_2 = (1, 2, 3)$ , a point lesson 3.5.1 found in the plane.

The residual is $\mathbf{r} = (0, 1, 4) - (1, 2, 3) = (-1, -1, 1)$ . Check: $\mathbf{a}_1 \cdot \mathbf{r} = -1 + 0 + 1 = 0$ and $\mathbf{a}_2 \cdot \mathbf{r} = 0 - 1 + 1 = 0$ . The distance from $\mathbf{b}$ to the plane is $\lVert \mathbf{r} \rVert = \sqrt{3} \approx 1.7321$ , and Pythagoras holds again: $17 = 14 + 3$ .

The projection of $\mathbf{b}$ onto a plane. The residual $\mathbf{r}$ is orthogonal to both columns, so it meets the plane at a right angle, and $\mathbf{p}$ is the nearest point.

Other points of the plane are further away. $A(1.5, 2) = (1.5, 2, 3.5)$ is 1.8708 from $\mathbf{b}$ , and $A(1, 1) = (1, 1, 2)$ is 2.2361 away.

In $\mathbb{R}^3$ , a plane through the origin leaves only one direction at right angles to it, here that of $(1, 1, -1)$ . So every residual for this plane is a multiple of $(1, 1, -1)$ . A point already in the plane, such as $(2, 3, 5)$ , is its own projection, with residual $\mathbf{0}$ .

A^\top A

import numpy as np A = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) b = np.array([0.0, 1.0, 4.0]) print(A.T @ A, A.T @ b) x_hat = np.linalg.solve(A.T @ A, A.T @ b) p = A @ x_hat r = b - p print(x_hat, p, r, A.T @ r, round(np.linalg.norm(r), 4)) for x in [(1, 2), (1.5, 2), (1, 1), (0, 0)]: print(x, A @ np.array(x), round(np.linalg.norm(b - A @ np.array(x)), 4)) for point in [(3.0, 1.0, 1.0), (2.0, 3.0, 5.0)]: point = np.array(point) p = A @ np.linalg.solve(A.T @ A, A.T @ point) print(point, p, point - p)

⌘/Ctrl + Enter runs the cell

The projection matrix

The formula $\mathbf{p} = A(A^\top A)^{-1}A^\top\mathbf{b}$ multiplies $\mathbf{b}$ by one matrix, the same for every $\mathbf{b}$ . It is the projection matrix $P$ onto the column space.

Onto the line through $\mathbf{u}$ : $P = \frac{\mathbf{u}\mathbf{u}^\top}{\mathbf{u}^\top\mathbf{u}}$ , a column times a row (lesson 3.2.3). For $\mathbf{u} = (3, 1)$ , $P = \frac{1}{10}\begin{pmatrix} 9 & 3 \\ 3 & 1 \end{pmatrix}$ , and $P\mathbf{v} = (1.5, 0.5)$ .
Onto an axis: for $\mathbf{u} = \mathbf{e}_1$ , $P = \begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}$ , the projection of lesson 3.2.2.
Onto the plane above: $P = \frac{1}{3}\begin{pmatrix} 2 & -1 & 1 \\ -1 & 2 & 1 \\ 1 & 1 & 2 \end{pmatrix}$ , and $P\mathbf{b} = (1, 2, 3)$ .

Projecting twice changes nothing: $\mathbf{p}$ is already in the plane, so its projection is itself. In symbols, $P^2 = P$ . Every projection matrix is also symmetric, and $I - P$ gives the residual, $(I - P)\mathbf{b} = \mathbf{r}$ .

Rule: Projection matrix

\begin{gathered} P = A(A^\top A)^{-1}A^\top \\[4pt] P^2 = P, \quad P^\top = P \end{gathered}

$P\mathbf{b}$ is the projection of $\mathbf{b}$ and $(I - P)\mathbf{b}$ its residual. A projection flattens space onto the column space, so unless that is all of $\mathbb{R}^n$ , $P$ has no inverse (lesson 3.2.3).

P

import numpy as np u = np.array([3.0, 1.0]) P_line = np.outer(u, u) / (u @ u) print(np.round(10 * P_line, 10), P_line @ np.array([1.0, 2.0])) A = np.array([[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]) P = A @ np.linalg.inv(A.T @ A) @ A.T b = np.array([0.0, 1.0, 4.0]) print(np.round(3 * P, 10)) print(np.round(P @ b, 10), np.round((np.eye(3) - P) @ b, 10)) print(np.allclose(P @ P, P), np.allclose(P, P.T), np.linalg.matrix_rank(P))

⌘/Ctrl + Enter runs the cell

Orthonormal columns

Vectors are orthonormal when each has length 1 and every pair is orthogonal. Write $Q$ for a matrix whose columns $\mathbf{q}_1, \ldots, \mathbf{q}_k$ are orthonormal. Entry $(i, j)$ of $Q^\top Q$ is $\mathbf{q}_i \cdot \mathbf{q}_j$ , which is 1 when $i = j$ and 0 otherwise, so $Q^\top Q = I$ .

Then $(A^\top A)^{-1}$ in the projection formula is $I^{-1} = I$ , and nothing needs solving.

Rule: Projection with orthonormal columns

\begin{gathered} Q^\top Q = I \\[4pt] \mathbf{p} = QQ^\top\mathbf{b} \\[4pt] = (\mathbf{q}_1 \cdot \mathbf{b})\,\mathbf{q}_1 + \cdots + (\mathbf{q}_k \cdot \mathbf{b})\,\mathbf{q}_k \end{gathered}

With orthonormal columns, the projection onto the plane is the sum of the projections onto each line $\mathbf{q}_i$ . That is lesson 3.5.1's shortcut for coordinates in an orthogonal basis: each coordinate is a dot product.

Making the columns orthonormal. The plane above has columns that are not orthogonal: $\mathbf{a}_1 \cdot \mathbf{a}_2 = 1$ . Keep the direction of $\mathbf{a}_1$ and normalise it: $\mathbf{q}_1 = \frac{1}{\sqrt{2}}(1, 0, 1)$ .

Then take from $\mathbf{a}_2$ its projection onto $\mathbf{a}_1$ . The residual $\mathbf{a}_2 - \frac{1}{2}\mathbf{a}_1 = (-0.5, 1, 0.5)$ is orthogonal to $\mathbf{a}_1$ and still in the plane. Normalised, it is $\mathbf{q}_2 \approx (-0.4082, 0.8165, 0.4082)$ .

For $\mathbf{b} = (0, 1, 4)$ , the two dot products are $\mathbf{q}_1 \cdot \mathbf{b} \approx 2.8284$ and $\mathbf{q}_2 \cdot \mathbf{b} \approx 2.4495$ . So $\mathbf{p} = 2.8284\,\mathbf{q}_1 + 2.4495\,\mathbf{q}_2 = (1, 2, 3)$ , as before.

$Q$ here is $3 \times 2$ , so $Q^\top Q = I$ but $QQ^\top$ is not $I$ : it is $P$ . np.linalg.qr builds orthonormal columns for any independent columns; their signs can differ between computers, but $QQ^\top$ cannot. Lesson 3.6.3 meets square $Q$ , for which $Q^\top Q = QQ^\top = I$ .

\mathbf{q}_1

import numpy as np a1 = np.array([1.0, 0.0, 1.0]) a2 = np.array([0.0, 1.0, 1.0]) q1 = a1 / np.linalg.norm(a1) w = a2 - (a1 @ a2) / (a1 @ a1) * a1 q2 = w / np.linalg.norm(w) Q = np.column_stack([q1, q2]) print(w, np.round(q2, 4)) print(np.round(Q.T @ Q, 10) + 0.0) b = np.array([0.0, 1.0, 4.0]) print(np.round(Q.T @ b, 4), np.round(Q @ Q.T @ b, 10) + 0.0) A = np.column_stack([a1, a2]) P = A @ np.linalg.inv(A.T @ A) @ A.T Q_qr, _ = np.linalg.qr(A) print(np.allclose(Q @ Q.T, P), np.allclose(Q_qr @ Q_qr.T, P))

⌘/Ctrl + Enter runs the cell

Lesson 3.5.3 uses the projection onto a column space to solve systems that have no solution.

Common mistakes

The coefficient divides by $\mathbf{u} \cdot \mathbf{u}$ , not by $\lVert \mathbf{u} \rVert$ . Dividing by the length once gives the shadow's signed length of lesson 3.1.2, a number; the projection is a vector.
The order matters. $\mathbf{v} = (1, 2)$ onto the line through $\mathbf{u} = (3, 1)$ is $(1.5, 0.5)$ , but $\mathbf{u}$ onto the line through $\mathbf{v}$ is $(1, 2)$ .
The residual is measured at right angles, not straight down. The point of the plane straight below $(0, 1, 4)$ is $(0, 1, 1)$ , at distance 3; the nearest is $(1, 2, 3)$ , at $\sqrt{3}$ .
Projecting onto each column and adding works only for orthogonal columns. For $\mathbf{a}_1$ and $\mathbf{a}_2$ above, the separate projections of $\mathbf{b}$ add up to $(2, 2.5, 4.5)$ , not $(1, 2, 3)$ .
$Q^\top Q = I$ , but $QQ^\top = I$ only for a square $Q$ . For a tall $Q$ , $QQ^\top$ is the projection matrix.
Check the right angle with a tolerance. A residual's dot product with a column can compute as $10^{-16}$ rather than 0 (lesson 3.1.2).

Orthogonal projection

0 of 12 correct

1.
Project $\mathbf{v} = (2, 3)$ onto the line through $\mathbf{u} = (1, 1)$ . What is the coefficient of $\mathbf{u}$ ?
2.
Project $(1, 7)$ onto the line through $(3, 4)$ . What is the second component of the projection?
3.
What is the distance from the point $(1, 7)$ to the line through $(3, 4)$ ?
4.
Project $(6, -2)$ onto the line through $(1, 3)$ . What is the length of the projection?
5.
Project $(3, 3, 0)$ onto the line through $(1, 2, 2)$ . What is the third component of the residual?
6.
The projection of $\mathbf{v}$ onto the line through $\mathbf{u} = (3, 1)$ is $(1.5, 0.5)$ . What is the first component of its projection onto the line through $(-6, -2)$ ?
7.
The squared distance from $\mathbf{v}$ to the point $t\,\mathbf{u}$ is $5 - 10t + 10t^2$ . At which $t$ is it smallest?
8.
$P = \frac{\mathbf{u}\mathbf{u}^\top}{\mathbf{u}^\top\mathbf{u}}$ for $\mathbf{u} = (1, 2)$ . Find the entry in row 1, column 2
9.
$P$ is a projection matrix and $P\mathbf{b} = (2, 0, 2)$ . What is the third component of $P(P\mathbf{b})$ ?
10.
Project $(3, 1, 1)$ onto the plane spanned by $(1, 0, 1)$ and $(0, 1, 1)$ . What is the first component of the projection?
11.
$Q$ has orthonormal columns $\mathbf{q}_1$ , $\mathbf{q}_2$ with $\mathbf{q}_1 \cdot \mathbf{b} = 3$ and $\mathbf{q}_2 \cdot \mathbf{b} = 4$ . What is the length of the projection $QQ^\top\mathbf{b}$ ?
12.
$Q$ is $5 \times 2$ with orthonormal columns. What is the entry in row 2, column 1 of $Q^\top Q$ ?

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\mathbf{v}

Orthogonal projection

The shadow as a vector

Rule: Projection onto a line

The residual is orthogonal

The nearest point on the line

Rule: The nearest point

Projection onto a plane

Rule: Projection onto a column space

A worked example

The projection matrix

Rule: Projection matrix

Orthonormal columns

Rule: Projection with orthonormal columns

Common mistakes

Orthogonal projection

Summary card

Free further reading