3.6.3 Symmetric matrices and the spectral theorem

In this lesson

What symmetry guarantees: real eigenvalues and perpendicular eigenvectors
Unit eigenvectors as the columns of an orthogonal matrix $Q$ , with $Q^\top Q = I$
The spectral theorem, $A = Q\Lambda Q^\top$ , and its picture: the unit circle becomes an ellipse along the eigenvectors
A symmetric matrix as a sum of projections, $\lambda_1\mathbf{q}_1\mathbf{q}_1^\top + \lambda_2\mathbf{q}_2\mathbf{q}_2^\top$
A $3 \times 3$ example, and a repeated eigenvalue
np.linalg.eigh, the tool for symmetric matrices, and its one trap

You need lessons 3.6.1 and 3.6.2, symmetric matrices and the transpose (lesson 3.2.3), perpendicular vectors (lesson 3.1.2), and orthonormal columns and projections (lesson 3.5.2).

What symmetry guarantees

The matrix $A = \begin{pmatrix} 1 & 2 \\ 3 & 2 \end{pmatrix}$ of lesson 3.6.1 has eigenvectors $(2, 3)$ and $(1, -1)$ , at an angle of 101.31° (lesson 3.1.2). The symmetric matrix $S = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ has eigenvectors $(1, 1)$ and $(1, -1)$ , at exactly 90°. That is no accident.

Rule: Symmetric matrices

A^\top = A \implies \lambda \text{ real}, \ \mathbf{q}_i \perp \mathbf{q}_j

Let $A$ be a symmetric $n \times n$ matrix with real entries (lesson 3.2.3). Its $n$ eigenvalues, counted with repeats, are all real. Eigenvectors of different eigenvalues are perpendicular, and $A$ always has $n$ eigenvectors that are perpendicular to each other, even when an eigenvalue repeats.

This is stated here without proof. For a $2 \times 2$ symmetric matrix $\begin{pmatrix} a & b \\ b & d \end{pmatrix}$ , the reality of the eigenvalues shows in the discriminant of lesson 3.6.1:

\begin{gathered} (a + d)^2 - 4(ad - b^2) \\ = (a - d)^2 + 4b^2 \end{gathered}

A sum of squares is never negative, so the eigenvalues are never complex. A symmetric matrix cannot turn the plane the way a rotation does. The discriminant is 0 only when $a = d$ and $b = 0$ , that is, for a multiple of $I$ .

A $3 \times 3$ example. Take

B = \begin{pmatrix} 3 & -1 & 1 \\ -1 & 5 & -1 \\ 1 & -1 & 3 \end{pmatrix}

It sends $(1, -2, 1)$ to $(6, -12, 6)$ , $(1, 1, 1)$ to $(3, 3, 3)$ and $(1, 0, -1)$ to $(2, 0, -2)$ . So its eigenvalues are 6, 3 and 2, and the three eigenvectors have dot products $1 - 2 + 1 = 0$ , $1 + 0 - 1 = 0$ and $1 + 0 - 1 = 0$ : each pair is perpendicular.

B

import numpy as np def angle(u, v): return round(float(np.degrees(np.arccos(u @ v / (np.linalg.norm(u) * np.linalg.norm(v))))), 2) print(angle(np.array([2, 3]), np.array([1, -1])), angle(np.array([1, 1]), np.array([1, -1]))) B = np.array([[3, -1, 1], [-1, 5, -1], [1, -1, 3]]) v1, v2, v3 = np.array([1, -2, 1]), np.array([1, 1, 1]), np.array([1, 0, -1]) for v in (v1, v2, v3): print(v, B @ v) print(v1 @ v2, v1 @ v3, v2 @ v3) a, b, d = 2, 1, 2 # S = [[a, b], [b, d]] print((a + d)**2 - 4 * (a * d - b * b), (a - d)**2 + 4 * b * b)

⌘/Ctrl + Enter runs the cell

The orthogonal matrix Q

Scale each eigenvector to length 1. For $S$ that gives

\begin{gathered} \mathbf{q}_1 = \tfrac{1}{\sqrt{2}}(1, 1) \approx (0.7071, 0.7071) \\[4pt] \mathbf{q}_2 = \tfrac{1}{\sqrt{2}}(1, -1) \approx (0.7071, -0.7071) \end{gathered}

The letter $\mathbf{q}$ marks an eigenvector of length 1. These two are perpendicular and of length 1: they are orthonormal, as in lesson 3.5.2.

Put them side by side as the columns of $Q$ . Entry $(i, j)$ of $Q^\top Q$ is $\mathbf{q}_i \cdot \mathbf{q}_j$ : 1 when $i = j$ , and 0 otherwise. So $Q^\top Q = I$ .

Rule: An orthogonal matrix

\begin{gathered} Q^\top Q = I \\[4pt] Q^{-1} = Q^\top \end{gathered}

A square matrix with orthonormal columns is called orthogonal. Its inverse is its transpose, so there is nothing to compute, and $QQ^\top = I$ as well. As a map it keeps every length and angle: it is a rotation or a reflection (lesson 3.2.2).

The spectral theorem

Lesson 3.6.2 wrote a diagonalisable matrix as $P\Lambda P^{-1}$ . For a symmetric matrix, take $P = Q$ ; then $P^{-1}$ is simply $Q^\top$ .

Rule: The spectral theorem

A = Q\Lambda Q^\top

Every symmetric matrix factors this way. $Q$ is orthogonal, with orthonormal eigenvectors as its columns, and $\Lambda$ holds the real eigenvalues in the same order. The set of eigenvalues is called the spectrum of $A$ , hence the name.

Read the product from right to left. $Q^\top\mathbf{x} = (\mathbf{q}_1 \cdot \mathbf{x},\ \mathbf{q}_2 \cdot \mathbf{x})$ : the coordinates of $\mathbf{x}$ along the eigenvectors are plain dot products. $\Lambda$ multiplies each coordinate by its eigenvalue, and $Q$ builds the vector back.

For $S$ and $\mathbf{x} = (2, 0)$ :

step	result
$Q^\top\mathbf{x}$ , the coordinates	$(1.4142, 1.4142)$
times $\Lambda = \begin{pmatrix} 3 & 0 \\ 0 & 1 \end{pmatrix}$	$(4.2426, 1.4142)$
times $Q$ , back to the plane	$(4, 2)$

That is $S(2, 0) = (4, 2)$ , the first column of $S$ doubled.

Q

import numpy as np S = np.array([[2, 1], [1, 2]]) q1 = np.array([1, 1]) / np.sqrt(2) q2 = np.array([1, -1]) / np.sqrt(2) Q = np.column_stack([q1, q2]) Lam = np.diag([3, 1]) print(np.round(Q, 4)) print(np.round(Q.T @ Q, 10) + 0.0) print(np.allclose(Q @ Lam @ Q.T, S)) x = np.array([2, 0]) c = Q.T @ x # coordinates along q1 and q2 print(np.round(c, 4), np.round(Lam @ c, 4), np.round(Q @ (Lam @ c), 6) + 0.0, S @ x)

⌘/Ctrl + Enter runs the cell

The picture: a circle becomes an ellipse

Each unit vector $(\cos t, \sin t)$ is a point of the unit circle (lesson 1.7.1). $Q^\top$ keeps lengths, so it moves the circle onto itself. $\Lambda$ then stretches it by $\lambda_1$ along one axis and by $\lambda_2$ along the other, and $Q$ turns those axes onto $\mathbf{q}_1$ and $\mathbf{q}_2$ .

So a symmetric matrix maps the unit circle to an ellipse whose axes point along its eigenvectors. The half-lengths of the axes, the semi-axes, are $\lvert\lambda_1\rvert$ and $\lvert\lambda_2\rvert$ .

$S$ stretches the unit circle by 3 along $\mathbf{q}_1$ and by 1 along $\mathbf{q}_2$ : an ellipse whose axes are the perpendicular eigenvectors. Every other unit vector, such as $(1, 0)$ , lands somewhere between.

No unit vector is stretched more than $\lvert\lambda_1\rvert$ or less than $\lvert\lambda_2\rvert$ . Checking 720 points round the circle, the images of $S$ have lengths between exactly 1 and 3, reached along $\mathbf{q}_2$ and $\mathbf{q}_1$ .

A negative eigenvalue flips one axis but gives the same ellipse. $N = \begin{pmatrix} 1 & 2 \\ 2 & 1 \end{pmatrix}$ has the same eigenvectors as $S$ , with eigenvalues 3 and $-1$ : it sends $\mathbf{q}_2$ to $-\mathbf{q}_2$ .

A matrix that is not symmetric also maps the circle to an ellipse, but the axes of that ellipse are not its eigenvectors. Lesson 3.8.1 finds them with the singular value decomposition.

S

import numpy as np S = np.array([[2, 1], [1, 2]]) t = np.linspace(0, 2 * np.pi, 721) # 720 steps round the circle circle = np.vstack([np.cos(t), np.sin(t)]) lengths = np.linalg.norm(S @ circle, axis=0) print(round(lengths.max(), 6), round(lengths.min(), 6)) q1 = np.array([1, 1]) / np.sqrt(2) q2 = np.array([1, -1]) / np.sqrt(2) print(np.round(S @ q1, 4), np.round(S @ q2, 4)) N = np.array([[1, 2], [2, 1]]) # eigenvalues 3 and -1 print(np.round(N @ q1, 4), np.round(N @ q2, 4))

⌘/Ctrl + Enter runs the cell

A sum of projections

Multiply out $Q\Lambda Q^\top$ one column of $Q$ at a time: each eigenvector contributes its own piece.

Rule: The spectral decomposition

A = \lambda_1\mathbf{q}_1\mathbf{q}_1^\top + \lambda_2\mathbf{q}_2\mathbf{q}_2^\top

Each $\mathbf{q}\mathbf{q}^\top$ , a column times a row, is the matrix that projects onto the line through $\mathbf{q}$ (lesson 3.5.2). An $n \times n$ symmetric matrix is a sum of $n$ such pieces.

For $S$ , the two pieces are

\begin{gathered} \mathbf{q}_1\mathbf{q}_1^\top = \begin{pmatrix} 0.5 & 0.5 \\ 0.5 & 0.5 \end{pmatrix} \\[6pt] \mathbf{q}_2\mathbf{q}_2^\top = \begin{pmatrix} 0.5 & -0.5 \\ -0.5 & 0.5 \end{pmatrix} \end{gathered}

and $3\,\mathbf{q}_1\mathbf{q}_1^\top + 1\,\mathbf{q}_2\mathbf{q}_2^\top = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix} = S$ .

So $A\mathbf{x} = \lambda_1(\mathbf{q}_1 \cdot \mathbf{x})\,\mathbf{q}_1 + \lambda_2(\mathbf{q}_2 \cdot \mathbf{x})\,\mathbf{q}_2$ : project $\mathbf{x}$ onto each eigenvector line, scale by the eigenvalue, and add. For $\mathbf{x} = (2, 0)$ the first projection is $(1, 1)$ , so the first piece gives $(3, 3)$ and the second $(1, -1)$ : together $(4, 2)$ .

The same holds for $B$ , with three pieces. Dropping the pieces with the smallest eigenvalues leaves a simpler matrix that still does most of what $A$ does. Case study 3.6.5 does this with a covariance matrix, and lesson 3.8.2 does it for any matrix.

\mathbf{q}\mathbf{q}^\top

import numpy as np q1 = np.array([1, 1]) / np.sqrt(2) q2 = np.array([1, -1]) / np.sqrt(2) P1, P2 = np.outer(q1, q1), np.outer(q2, q2) # q q^T, a column times a row print(np.round(P1, 4)) print(np.round(P2, 4)) print(np.round(3 * P1 + 1 * P2, 6) + 0.0) print(np.allclose(P1 @ P1, P1), np.round(P1 @ np.array([2, 0]), 4)) B = np.array([[3, -1, 1], [-1, 5, -1], [1, -1, 3]]) pieces = [(6, np.array([1, -2, 1])), (3, np.array([1, 1, 1])), (2, np.array([1, 0, -1]))] total = sum(lam * np.outer(v, v) / (v @ v) for lam, v in pieces) print(np.round(total, 6) + 0.0)

⌘/Ctrl + Enter runs the cell

eigh: eigenvalues of a symmetric matrix

np.linalg.eigh(A) is made for symmetric matrices. It returns the eigenvalues as real numbers in increasing order, and a matrix $Q$ whose columns are orthonormal eigenvectors.

Prefer it to eig whenever the matrix is symmetric:

It keeps the promises of this lesson. The eigenvalues are always real and the columns always orthonormal, even when an eigenvalue repeats. eig guarantees neither.
It is faster and more accurate, because it uses the symmetry.

Reverse its output to put the largest eigenvalue first, and fix the signs as in lesson 3.6.1. For $B$ it gives 6, 3 and 2, with the unit eigenvectors $(1, -2, 1)/\sqrt{6}$ , $(1, 1, 1)/\sqrt{3}$ and $(1, 0, -1)/\sqrt{2}$ .

The rule of lesson 3.6.1 for $n \times n$ matrices checks out. The eigenvalues of $B$ add up to $6 + 3 + 2 = 11$ , its trace $3 + 5 + 3$ , and multiply to 36, its determinant.

\sqrt{6}

import numpy as np B = np.array([[3, -1, 1], [-1, 5, -1], [1, -1, 3]]) values, Q = np.linalg.eigh(B) print(np.round(values, 6)) # increasing order values, Q = values[::-1], Q[:, ::-1] # largest first Q = Q * np.sign(Q[0]) # first entry of each column positive print(np.round(values, 6), np.round(Q, 4) + 0.0) print(np.round(Q * np.sqrt([6, 3, 2]), 6) + 0.0) print(np.allclose(Q.T @ Q, np.eye(3)), np.allclose(Q @ np.diag(values) @ Q.T, B)) print(round(values.sum(), 6), np.trace(B), round(values.prod(), 6), round(np.linalg.det(B), 6))

⌘/Ctrl + Enter runs the cell

A repeated eigenvalue

$J = \begin{pmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{pmatrix}$ has eigenvalues 4, 1 and 1. The eigenvector of 4 is $(1, 1, 1)$ . For the eigenvalue 1, every vector perpendicular to $(1, 1, 1)$ works, such as $(1, -1, 0)$ or $(1, 1, -2)$ : a whole plane of eigenvectors.

eigh picks two perpendicular unit vectors in that plane. Which two it picks can differ between computers, so compare the eigenvalues and $Q^\top Q = I$ , not the columns themselves.

The trap: eigh trusts you

eigh assumes the matrix is symmetric and reads only the entries on and below the diagonal. Given $T = \begin{pmatrix} 2 & 5 \\ 1 & 2 \end{pmatrix}$ , it returns 1 and 3, the eigenvalues of $\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ . The true eigenvalues of $T$ are 4.2361 and $-0.2361$ .

So check symmetry first with np.allclose(A, A.T), as case study 3.2.5 did for a covariance matrix.

Covariance matrices

The covariance matrix $\Sigma$ of case study 3.2.5 is symmetric, so all of this applies to it: real eigenvalues and perpendicular eigenvectors. Its eigenvalues are 0.0751, 0.0225 and 0.0117, and they add up to its trace, 0.1093, the sum of the three variances.

Case study 3.6.5 reads such eigenvalues as the variance along each eigenvector. Module 3.7 uses their signs to tell which symmetric matrices can be covariance matrices at all, starting with quadratic forms in lesson 3.7.1.

J

import numpy as np J = np.array([[2, 1, 1], [1, 2, 1], [1, 1, 2]]) # the eigenvalue 1 repeats values, Q = np.linalg.eigh(J) print(np.round(values, 6), np.allclose(Q.T @ Q, np.eye(3))) print(J @ np.array([1, -1, 0]), J @ np.array([1, 1, -2])) T = np.array([[2, 5], [1, 2]]) # not symmetric print(np.round(np.linalg.eigh(T).eigenvalues, 4)) print(np.round(np.sort(np.linalg.eig(T).eigenvalues.real)[::-1], 4)) print(np.allclose(T, T.T)) sigma = np.array([0.12, 0.18, 0.25]) R = np.array([[1, 0.3, 0.1], [0.3, 1, 0.5], [0.1, 0.5, 1]]) Sigma = np.diag(sigma) @ R @ np.diag(sigma) # case study 3.2.5 values = np.linalg.eigh(Sigma).eigenvalues[::-1] print(np.round(values, 4), round(values.sum(), 4), round(np.trace(Sigma), 4))

⌘/Ctrl + Enter runs the cell

Common mistakes

Perpendicular eigenvectors need symmetry. $A = \begin{pmatrix} 1 & 2 \\ 3 & 2 \end{pmatrix}$ has eigenvectors at 101.31°.
Normalise before you build $Q$ . With $(1, 1)$ and $(1, -1)$ as columns, $Q^\top Q = 2I$ , and $Q^\top$ is not the inverse.
eigh lists the eigenvalues in increasing order. The largest is last; reverse both the values and the columns.
eigh reads only one triangle. Check np.allclose(A, A.T) before you trust it.
The semi-axes are $\lvert\lambda\rvert$ . A negative eigenvalue flips its axis but does not shrink the ellipse.
A repeated eigenvalue gives a whole plane of eigenvectors. Any perpendicular pair in it is correct, so do not compare those columns between computers.

Symmetric matrices and the spectral theorem

0 of 12 correct

1.
Find the larger eigenvalue of the symmetric matrix $\begin{pmatrix} 5 & 2 \\ 2 & 2 \end{pmatrix}$
2.
That matrix has the eigenvector $(2, 1)$ for $\lambda = 6$ . Its other eigenvector is $(1, y)$ . Find $y$ without solving any equations
3.
$\mathbf{q}_1 = (0.6, 0.8)$ . The unit vector $\mathbf{q}_2 = (-0.8, y)$ is perpendicular to it. Find $y$
4.
$Q$ is a $3 \times 3$ orthogonal matrix. What is the entry in row 1, column 2 of $Q^\top Q$ ?
5.
$Q$ is orthogonal and $\lVert\mathbf{x}\rVert = 5$ . What is $\lVert Q\mathbf{x}\rVert$ ?
6.
A symmetric matrix has eigenvalues $-5$ and 2. What is the longest semi-axis of the image of the unit circle?
7.
The unit circle has area $\pi$ . A symmetric matrix with eigenvalues 3 and 2 maps it to an ellipse. What is the ellipse's area? (type pi for $\pi$ )
8.
$A = 5\,\mathbf{q}_1\mathbf{q}_1^\top + 2\,\mathbf{q}_2\mathbf{q}_2^\top$ with $\mathbf{q}_1 = (0.6, 0.8)$ and $\mathbf{q}_2 = (-0.8, 0.6)$ . Find the top-left entry of $A$
9.
For the same $A$ , find the first entry of $A\mathbf{q}_1$
10.
np.linalg.eigh returns the eigenvalues $[0.5, 2, 7]$ . At which position, 1, 2 or 3, is the eigenvector of the largest one?
11.
A symmetric $3 \times 3$ matrix has trace 12, and two of its eigenvalues are 2 and 4. Find the third
12.
eigh reads only the entries on and below the diagonal. Given $\begin{pmatrix} 2 & 5 \\ 1 & 2 \end{pmatrix}$ , what is the larger eigenvalue it returns?

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

n

Symmetric matrices and the spectral theorem

What symmetry guarantees

Rule: Symmetric matrices

The orthogonal matrix Q

Rule: An orthogonal matrix

The spectral theorem

Rule: The spectral theorem

The picture: a circle becomes an ellipse

A sum of projections

Rule: The spectral decomposition

eigh: eigenvalues of a symmetric matrix

A repeated eigenvalue

The trap: eigh trusts you

Covariance matrices

Common mistakes

Symmetric matrices and the spectral theorem

Summary card

Free further reading