Transpose, identity and inverse

The transpose

The transpose of a matrix $A$, written $A^\top$, turns its rows into columns: row $i$ of $A$ becomes column $i$ of $A^\top$. So the entry in row $i$, column $j$ of $A^\top$ is $a_{ji}$, and an $m \times n$ matrix has an $n \times m$ transpose.

Transposing twice gives back the matrix you started with: $(A^\top)^\top = A$. A column vector turned on its side is a row: for $\mathbf{u} = (3, 1)$, $\mathbf{u}^\top = (3\ \ 1)$, a $1 \times 2$ matrix.

The transpose of a product

With $A$ and $C$ from lesson 3.2.1, $AC = \begin{pmatrix} 4 & 1 \\ 5 & 3 \end{pmatrix}$, so $(AC)^\top = \begin{pmatrix} 4 & 5 \\ 1 & 3 \end{pmatrix}$. That equals $C^\top A^\top$, with the order reversed. The other order, $A^\top C^\top = \begin{pmatrix} 2 & 3 \\ 1 & 5 \end{pmatrix}$, does not match.

Here is why. Entry $(1, 2)$ of $(AC)^\top$ is entry $(2, 1)$ of $AC$: row 2 of $A$ dotted with column 1 of $C$, $(-1, 3) \cdot (1, 2) = 5$.

Those two vectors are column 2 of $A^\top$ and row 1 of $C^\top$. So the same dot product is entry $(1, 2)$ of $C^\top A^\top$.

Shapes say the same. $M$ is $2 \times 3$ and $N = \begin{pmatrix} 1 & 0 \\ 2 & 1 \\ 0 & 3 \end{pmatrix}$ is $3 \times 2$, so $MN$ and its transpose are $2 \times 2$. $N^\top M^\top$ is $(2 \times 3)(3 \times 2)$, also $2 \times 2$; but $M^\top N^\top$ is $(3 \times 2)(2 \times 3)$, a $3 \times 3$ matrix.

Symmetric matrices

A square matrix is symmetric when it equals its transpose, $A^\top = A$: every entry $a_{ij}$ equals $a_{ji}$. The matrix is a mirror image of itself across its main diagonal, the entries $a_{11}, a_{22}, \ldots$ from top left to bottom right.

Both are symmetric. $A = \begin{pmatrix} 2 & 1 \\ -1 & 3 \end{pmatrix}$ is not: $a_{12} = 1$ but $a_{21} = -1$.

Two ways always make a symmetric matrix. $M^\top M$ is symmetric for any $M$, because $(M^\top M)^\top = M^\top (M^\top)^\top = M^\top M$ by the rule above. And $A + A^\top$ is symmetric for any square $A$.

Case study 3.2.5 meets the most important symmetric matrix in finance, the covariance matrix.

The dot product as uᵀv

Lesson 3.2.1 found that a row times a column is a single number, their dot product. A column vector $\mathbf{u}$ with $n$ components is $n \times 1$, so $\mathbf{u}^\top$ is $1 \times n$ and $\mathbf{u}^\top\mathbf{v}$ is $1 \times 1$.

For $\mathbf{u} = (3, 1)$ and $\mathbf{v} = (1, 2)$, $\mathbf{u}^\top\mathbf{v} = 3 + 2 = 5$, and $\mathbf{u}^\top\mathbf{u} = 10$, so the length is $\lVert \mathbf{u} \rVert = \sqrt{10} \approx 3.162278$. The other order, $\mathbf{u}\mathbf{v}^\top$, is $(2 \times 1)(1 \times 2)$: the $2 \times 2$ table $\begin{pmatrix} 3 & 6 \\ 1 & 2 \end{pmatrix}$, not a number.

A row, a square matrix and a column multiply to a number too: $\mathbf{u}^\top S\mathbf{v}$ is $(1 \times 2)(2 \times 2)(2 \times 1)$. Here $S\mathbf{v} = (4, 5)$, so $\mathbf{u}^\top S\mathbf{v} = 12 + 5 = 17$. Case study 3.2.5 writes a portfolio's variance in exactly this form.

The identity matrix

The identity matrix $I$ is square, with 1 at every place on the main diagonal and 0 everywhere else. $I_2$ and $I_3$ are its $2 \times 2$ and $3 \times 3$ versions:

As a map it changes nothing: its columns are $\mathbf{e}_1$ and $\mathbf{e}_2$ themselves, so every vector stays put (lesson 3.2.2). It plays the part that 1 plays for numbers.

The inverse

In lesson 3.2.1, $A = \begin{pmatrix} 2 & 1 \\ -1 & 3 \end{pmatrix}$ sent $\mathbf{x} = (2, 1)$ to $(5, 1)$. Run backwards: which $\mathbf{x}$ does $A$ send to $(5, 1)$? The map that answers it for every point is the inverse of $A$, written $A^{-1}$.

The 2 × 2 formula

For a $2 \times 2$ matrix the inverse has a formula. Swap the two diagonal entries, change the sign of the other two, and divide by the number $ad - bc$.

Three examples:

• $B = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix}$ has $ad - bc = 2 - 1 = 1$, so $B^{-1} = \begin{pmatrix} 1 & -1 \\ -1 & 2 \end{pmatrix}$.
• $G = \begin{pmatrix} 4 & 2 \\ 1 & 3 \end{pmatrix}$ has $ad - bc = 12 - 2 = 10$, so $G^{-1} = \frac{1}{10}\begin{pmatrix} 3 & -2 \\ -1 & 4 \end{pmatrix} = \begin{pmatrix} 0.3 & -0.2 \\ -0.1 & 0.4 \end{pmatrix}$.
• $A$ has $ad - bc = 6 - (-1) = 7$, so $A^{-1} = \frac{1}{7}\begin{pmatrix} 3 & -1 \\ 1 & 2 \end{pmatrix}$. It sends $(5, 1)$ to $\frac{1}{7}(14, 7) = (2, 1)$, the point we started from.

Check $B$ entry by entry. Row 1 of $B$ dotted with column 1 of $B^{-1}$ is $2 \cdot 1 + 1 \cdot (-1) = 1$, and with column 2 it is $2 \cdot (-1) + 1 \cdot 2 = 0$. Row 2 gives $1 - 1 = 0$ and $-1 + 2 = 1$: together, the identity.

Undoing the maps of lesson 3.2.2

Each map in the gallery of lesson 3.2.2 has a natural undo, and the formula agrees with it.

• Shear $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$: shear back, $\begin{pmatrix} 1 & -1 \\ 0 & 1 \end{pmatrix}$.
• Stretch $\begin{pmatrix} 2 & 0 \\ 0 & 0.5 \end{pmatrix}$: shrink back, $\begin{pmatrix} 0.5 & 0 \\ 0 & 2 \end{pmatrix}$.
• Quarter turn $\begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$: turn back, $\begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix}$. That is also its transpose: for every rotation, $R^{-1} = R^\top$.
• Reflection $\begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$: reflect again. It is its own inverse.
• Projection $\begin{pmatrix} 1 & 0 \\ 0 & 0 \end{pmatrix}$: no undo. Its $ad - bc$ is $1 \cdot 0 - 0 \cdot 0 = 0$.

Undoing two maps

To undo "shear, then turn", undo the turn first and the shear second: the last step is the first one undone. With the shear $H = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ and the quarter turn $R = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$, $RH$ means shear first, so its inverse is $H^{-1}R^{-1}$.

Follow $(3, 2)$. The shear sends it to $(5, 2)$ and the turn to $(-2, 5)$.

Applying $H^{-1}R^{-1}$ brings $(-2, 5)$ back to $(3, 2)$. The wrong order, $R^{-1}H^{-1}$, sends it to $(5, 7)$ instead.

When there is no inverse

A $2 \times 2$ matrix with $ad - bc = 0$ has no inverse. Take

with $ad - bc = 1 - 1 = 0$. Its second column, $(2, 1)$, is twice its first, $(1, 0.5)$. So every image $Z\mathbf{x} = x_1(1, 0.5) + x_2(2, 1)$ is a multiple of $(1, 0.5)$: the whole plane is flattened onto the line $x_2 = x_1/2$.

A flattened plane cannot be unflattened. The points $(2, 0)$, $(0, 1)$, $(1, 0.5)$ and $(4, -1)$ all land on $(2, 1)$, so no map can send $(2, 1)$ back to a single start. The vector $(2, -1)$ even lands on $(0, 0)$, alongside $\mathbf{0}$ itself.

A matrix with no inverse is called singular. Lesson 3.4.1 calls $ad - bc$ the determinant and reads it as the factor by which the map scales areas, so 0 means flattened. Lesson 3.3.2 finds the same failure in larger matrices.

Solving Ax = b in NumPy

To find $\mathbf{x}$ with $A\mathbf{x} = \mathbf{b}$, multiply both sides by $A^{-1}$: $\mathbf{x} = A^{-1}\mathbf{b}$. That is the right idea, but not the right code.

np.linalg.solve(A, b) finds $\mathbf{x}$ without building $A^{-1}$. It does less arithmetic, and so it usually makes smaller rounding errors, which matters for large or nearly singular systems. Lesson 3.3.1 shows the method it uses, elimination.

For $A$ and $\mathbf{b} = (5, 1)$, both give $(2, 1)$. For the $3 \times 3$ system $L\mathbf{x} = \mathbf{c}$,

solve gives $\mathbf{x} \approx (1.2222, 0.5556, 2.1111)$, and multiplying back by $L$ gives $\mathbf{c}$. An exactly singular matrix such as $Z$ makes solve raise the same LinAlgError as inv.

Common mistakes

1. The order reverses. $(AB)^\top = B^\top A^\top$ and $(AB)^{-1} = B^{-1}A^{-1}$.
2. $A^{-1}$ is not the reciprocal of each entry. For $B = \begin{pmatrix} 2 & 1 \\ 1 & 1 \end{pmatrix}$ the reciprocals give $\begin{pmatrix} 0.5 & 1 \\ 1 & 1 \end{pmatrix}$, but $B^{-1} = \begin{pmatrix} 1 & -1 \\ -1 & 2 \end{pmatrix}$.
3. In the $2 \times 2$ formula, swap $a$ and $d$ and change the signs of $b$ and $c$; do not swap $b$ and $c$. Then divide by $ad - bc$.
4. Check $ad - bc$ first. If it is 0, there is no inverse: the map flattens the plane.
5. $\mathbf{u}^\top\mathbf{v}$ is a number; $\mathbf{u}\mathbf{v}^\top$ is a matrix.
6. Only square matrices have inverses, and in code, solve systems with np.linalg.solve rather than inv.