4.3.1 The Hessian and second-order Taylor expansion

In this lesson

The Hessian $H$ : the second partial derivatives of a function in one symmetric matrix, and what each entry measures
How $H$ moves the gradient when the point moves
The second-order Taylor expansion, $f(\mathbf{a} + \mathbf{h}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a}) \cdot \mathbf{h} + \frac{1}{2}\mathbf{h}^\top H\mathbf{h}$ , and its quadratic form
How fast its error falls, in every direction, and where the copy fails
The second derivative along a direction, $\mathbf{u}^\top H\mathbf{u}$
Computing $H$ in NumPy with the central differences of lesson 4.1.2

You need second partial derivatives from lesson 4.1.2, the gradient and the total differential from lessons 4.2.1 and 4.2.2, the quadratic copy from lesson 2.7.1, and quadratic forms from lesson 3.7.1.

The Hessian

A function of two variables has four second partial derivatives (lesson 4.1.2). $f_{xx}$ differentiates twice in $x$ with $y$ held fixed, $f_{yy}$ twice in $y$ , and $f_{xy}$ and $f_{yx}$ once in each. The Hessian puts them in a matrix.

Rule: The Hessian

H = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{yx} & f_{yy} \end{pmatrix}

For $n$ variables $x_1, \ldots, x_n$ , $H$ is $n \times n$ , and its entry in row $i$ , column $j$ is $\frac{\partial^2 f}{\partial x_i\,\partial x_j}$ . Since $f_{xy} = f_{yx}$ (lesson 4.1.2), $H$ is symmetric, so the tools of lessons 3.6.3 and 3.7.1 apply to it.

Take $f(x, y) = x^2y$ . Its first partials are $f_x = 2xy$ and $f_y = x^2$ . Differentiating again gives $f_{xx} = 2y$ , $f_{xy} = f_{yx} = 2x$ and $f_{yy} = 0$ . At the point $\mathbf{a} = (1, 2)$ :

\begin{gathered} f(\mathbf{a}) = 2, \quad \nabla f(\mathbf{a}) = \begin{pmatrix} 4 \\ 1 \end{pmatrix} \\[6pt] H = \begin{pmatrix} 4 & 2 \\ 2 & 0 \end{pmatrix} \end{gathered}

Each entry has a meaning. $f_{xx} = 4$ is the bend of the slice $y = 2$ , which is $2x^2$ (lessons 4.1.1 and 2.4.2). $f_{yy} = 0$ because the slice $x = 1$ is the straight line $f = y$ . And $f_{xy} = 2$ says how the slope in $x$ changes as $y$ moves: $f_x = 2xy$ rises by 2 per unit of $y$ at $x = 1$ .

The units follow. If $f$ is in dollars, $x$ in dollars and $y$ in years, then $f_{xy}$ is in dollars per dollar per year. Each term of the expansion below ends up in the units of $f$ .

H moves the gradient

Row 1 of $H$ holds the partials of $f_x$ , and row 2 those of $f_y$ . So the total differential of lesson 4.2.2, applied to $f_x$ and $f_y$ in turn, says how the whole gradient changes after a small step $\mathbf{h} = (h_1, h_2)$ .

Rule: H moves the gradient

\nabla f(\mathbf{a} + \mathbf{h}) \approx \nabla f(\mathbf{a}) + H\mathbf{h}

$H$ is to the gradient what the gradient is to $f$ : the change after a small step. Lesson 4.4.3 builds Newton's method on it.

For $x^2y$ , step from $(1, 2)$ to $(1.1, 2)$ , so $\mathbf{h} = (0.1, 0)$ and $H\mathbf{h} = (0.4, 0.2)$ . The rule predicts $\nabla f = (4.4, 1.2)$ . The true gradient is $(2 \times 1.1 \times 2,\ 1.1^2) = (4.4, 1.21)$ .

H

import numpy as np import sympy as sp x, y = sp.symbols("x y") f = x**2 * y print([sp.diff(f, v) for v in (x, y)]) print(sp.hessian(f, (x, y))) a = {x: 1, y: 2} print([sp.diff(f, v).subs(a) for v in (x, y)], sp.hessian(f, (x, y)).subs(a)) grad, H = np.array([4, 1]), np.array([[4, 2], [2, 0]]) h = np.array([0.1, 0]) print(grad + H @ h, [2 * 1.1 * 2, round(1.1**2, 6)])

⌘/Ctrl + Enter runs the cell

The second-order Taylor expansion

Lesson 2.7.1 copied a function of one variable near $a$ with a parabola, using $f''$ . The same copy works in several variables, with $H$ in place of $f''$ .

Walk from $\mathbf{a}$ along the step $\mathbf{h}$ : $g(t) = f(\mathbf{a} + t\mathbf{h})$ , a function of one variable $t$ . Its slope at $t = 0$ is $\nabla f(\mathbf{a}) \cdot \mathbf{h}$ , the chain rule of lesson 4.2.2. Applying the chain rule once more gives its second derivative, $\mathbf{h}^\top H\mathbf{h}$ ; a cell below checks this on an example.

The quadratic copy of $g$ about 0, evaluated at $t = 1$ , is $g(0) + g'(0) + \frac{1}{2}g''(0)$ . Since $g(1) = f(\mathbf{a} + \mathbf{h})$ , this is the rule.

Rule: Second-order Taylor expansion

\begin{gathered} f(\mathbf{a} + \mathbf{h}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a}) \cdot \mathbf{h} \\ + \tfrac{1}{2}\,\mathbf{h}^\top H\,\mathbf{h} \end{gathered}

The quadratic copy of $f$ near $\mathbf{a}$ . The step is $\mathbf{h}$ ; $\nabla f$ and $H$ are taken at $\mathbf{a}$ , so they are numbers. The first two terms are the tangent plane of lesson 4.2.2, and the last is a quadratic form (lesson 3.7.1).

Written out for two variables, with the quadratic form of lesson 3.7.1:

\begin{aligned} f(\mathbf{a} + \mathbf{h}) \approx{}& f + f_x h_1 + f_y h_2 \\ &+ \tfrac{1}{2}f_{xx}h_1^2 + f_{xy}h_1h_2 \\ &+ \tfrac{1}{2}f_{yy}h_2^2 \end{aligned}

Here $f$ , $f_x$ and the rest are their values at $\mathbf{a}$ . The cross term has no $\frac{1}{2}$ : the off-diagonal entry appears twice in $\mathbf{h}^\top H\mathbf{h}$ , as $2f_{xy}h_1h_2$ , and the $\frac{1}{2}$ cancels the 2.

Example 1: $x^2y$ at $(1, 2)$

With $f = 2$ , $\nabla f = (4, 1)$ and $H$ as above, the copy is

2 + 4h_1 + h_2 + 2h_1^2 + 2h_1h_2

step $\mathbf{h}$	$(0.1, 0.1)$	$(0.1, -0.2)$	$(-0.2, 0.1)$	$(0.2, 0.2)$	$(0.5, 0.5)$
$f(\mathbf{a} + \mathbf{h})$	2.541	2.178	1.344	3.168	5.625
tangent plane	2.5	2.2	1.3	3	4.5
quadratic copy	2.54	2.18	1.34	3.16	5.5
error of the copy	0.001	$-0.002$	0.004	0.008	0.125

Each error is $f$ minus the copy. The bend terms take the tangent plane's miss of 0.041 at $(0.1, 0.1)$ down to 0.001.

Algebra shows why. Expand the function at the step:

\begin{aligned} (1 + h_1)^2(2 + h_2) ={}& 2 + 4h_1 + h_2 \\ &+ 2h_1^2 + 2h_1h_2 \\ &+ h_1^2h_2 \end{aligned}

The first five terms are the copy. The error is exactly $h_1^2h_2$ , a term of degree 3: $0.1^2 \times 0.1 = 0.001$ , and $0.1^2 \times (-0.2) = -0.002$ , where the copy is too high.

f

import numpy as np def f(x, y): return x**2 * y a, fa = np.array([1, 2]), 2 grad, H = np.array([4, 1]), np.array([[4, 2], [2, 0]]) for h in [(0.1, 0.1), (0.1, -0.2), (-0.2, 0.1), (0.2, 0.2), (0.5, 0.5)]: h = np.array(h) plane = fa + grad @ h copy = plane + 0.5 * h @ H @ h true = f(*(a + h)) print(h, round(true, 6), round(plane, 6), round(copy, 6), round(true - copy, 6), round(h[0]**2 * h[1], 6))

⌘/Ctrl + Enter runs the cell

Example 2: $xe^y$ at $(2, 0)$

Here $f_x = e^y$ and $f_y = xe^y$ , then $f_{xx} = 0$ , $f_{xy} = e^y$ and $f_{yy} = xe^y$ (lesson 2.3.4 for $e^y$ ). At $(2, 0)$ , $e^0 = 1$ , so

\begin{gathered} f = 2, \quad \nabla f = \begin{pmatrix} 1 \\ 2 \end{pmatrix}, \quad H = \begin{pmatrix} 0 & 1 \\ 1 & 2 \end{pmatrix} \\[6pt] \text{copy: } 2 + h_1 + 2h_2 + h_1h_2 + h_2^2 \end{gathered}

At the step $(0.1, 0.1)$ the copy gives 2.32, against $2.1e^{0.1} = 2.320859$ . The tangent plane gives 2.3.

Level curves of $xe^y$ (solid), with those of its tangent plane and of its quadratic copy about $(2, 0)$ (dashed). The copy follows the bend of the curves near the point; further out they part.

How fast the error falls

Take steps in one direction, $\mathbf{h} = t\,(1, 1)$ , and shrink $t$ . For $xe^y$ at $(2, 0)$ :

$t$	0.4	0.2	0.1	shrinks by
error of the tangent plane	0.38	0.0871	0.0209	about 4
error of the quadratic copy	0.0604	0.00709	0.000859	about 8

The tangent plane's error falls like $t^2$ , as lesson 4.2.2 found. The copy's falls like $t^3$ , the first power it leaves out, as in lesson 2.7.1.

Rule: Error of the quadratic copy

f(\mathbf{a} + t\mathbf{u}) - \text{copy} \approx C\,t^3

For small steps $t\mathbf{u}$ in a fixed direction $\mathbf{u}$ . The constant $C$ depends on $f$ , on $\mathbf{a}$ and on the direction. Halve the step and the error shrinks about 8 times.

The error of the tangent plane and of the quadratic copy of $xe^y$ at $(2, 0)$ , on log scales. The slopes are 2 and 3: the bend terms add one to the power.

Both sides, and a direction with no bend

The error can have either sign. In example 1 it is $h_1^2h_2$ , which takes the sign of $h_2$ .

Along the direction $(-1, 1)$ in example 2, the bend term is $h_1h_2 + h_2^2 = -t^2 + t^2 = 0$ . There the copy is just the tangent plane. Its error still falls like $t^3$ : $-0.0131$ , $-0.00148$ and $-0.000175$ at $t = 0.4$ , 0.2 and 0.1, as when a Taylor coefficient is 0 in lesson 2.7.1.

t

import math import numpy as np def f(x, y): return x * math.exp(y) grad, H = np.array([1, 2]), np.array([[0, 1], [1, 2]]) plane = lambda h: 2 + grad @ h copy = lambda h: plane(h) + 0.5 * h @ H @ h errs = [] for t in [0.4, 0.2, 0.1]: h = t * np.array([1, 1]) true = f(2 + h[0], h[1]) errs.append((true - plane(h), true - copy(h))) print(t, f"{true - plane(h):.3g}", f"{true - copy(h):.3g}") print([round(float(errs[i][0] / errs[i + 1][0]), 2) for i in range(2)], [round(float(errs[i][1] / errs[i + 1][1]), 2) for i in range(2)]) for t in [0.4, 0.2, 0.1]: h = t * np.array([-1, 1]) true = f(2 + h[0], h[1]) print(t, f"{true - plane(h):.3g}", f"{true - copy(h):.3g}")

⌘/Ctrl + Enter runs the cell

The second derivative along a direction

The middle step of the derivation is worth keeping. For a unit vector $\mathbf{u}$ , the slice $g(t) = f(\mathbf{a} + t\mathbf{u})$ has second derivative $\mathbf{u}^\top H\mathbf{u}$ at $t = 0$ . It measures how the surface bends as you walk through $\mathbf{a}$ along $\mathbf{u}$ .

Rule: Second derivative along u

\frac{d^2}{dt^2} f(\mathbf{a} + t\mathbf{u}) \Big|_{t = 0} = \mathbf{u}^\top H\,\mathbf{u}

Along the axes it gives $f_{xx}$ and $f_{yy}$ . In other directions $f_{xy}$ joins in: $\mathbf{u}^\top H\mathbf{u} = f_{xx}u_1^2 + 2f_{xy}u_1u_2 + f_{yy}u_2^2$ .

For $xe^y$ at $(2, 0)$ , with $H = \begin{pmatrix} 0 & 1 \\ 1 & 2 \end{pmatrix}$ :

direction $\mathbf{u}$	$(1, 0)$	$(0, 1)$	$\frac{1}{\sqrt{2}}(1, 1)$	$\frac{1}{\sqrt{2}}(1, -1)$
$\mathbf{u}^\top H\mathbf{u}$	0	2	2	0

Along $x$ the slice is $xe^0 = x$ , a straight line, so the bend is 0. Along $y$ it is $2e^y$ , whose second derivative at 0 is 2. Lesson 4.3.2 reads the sign of $\mathbf{u}^\top H\mathbf{u}$ in every direction to tell a minimum from a saddle.

\mathbf{u}^\top H\mathbf{u}

import math import numpy as np H = np.array([[0, 1], [1, 2]]) r = 1 / math.sqrt(2) for u in [(1, 0), (0, 1), (r, r), (r, -r)]: u = np.array(u) print(np.round(u, 4), round(u @ H @ u, 6)) def g(t): # the slice of x e^y through (2, 0) along (r, r) return (2 + t * r) * math.exp(t * r) s = 0.001 print(round((g(s) - 2 * g(0) + g(-s)) / s**2, 4))

⌘/Ctrl + Enter runs the cell

Where the copy is exact, and where it fails

A quadratic is its own copy. For the tilted bowl $f(x, y) = x^2 + xy + y^2 - 3x$ of lesson 4.2.1, the Hessian is $\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ at every point, lesson 3.7.1's $S$ . The expansion has no terms of degree 3, so the copy is exact for any step, as $x^3$ about 2 was its own cubic copy in lesson 2.7.1.

About $(0, 0)$ , with $f = 0$ and $\nabla f = (-3, 0)$ , the step $(2, -1)$ gives $-6 + 3 = -3$ . That is $f(2, -1)$ exactly: the bottom of the bowl, found in lesson 4.2.1.

Far from the point the copy drifts. For $xe^y$ the step $(1, 1)$ gives $2 + 1 + 2 + 1 + 1 = 7$ , against $3e = 8.1548$ .

Small next to what? A step is small when it is small next to the distance over which $f$ bends, in each variable. A move of 1 may be tiny for one input and huge for another; case study 4.3.4 meets this with a stock price in dollars and a volatility in decimals.

No second derivatives, no copy. The cone $\sqrt{x^2 + y^2}$ has no gradient at the origin (lesson 4.2.1), as $|x|$ has no slope at 0. At such a point there is no tangent plane and no quadratic copy.

f(\mathbf{a})

import math import numpy as np def copy(fa, grad, H, h): h = np.array(h, dtype=float) return fa + np.array(grad) @ h + 0.5 * h @ np.array(H) @ h def bowl(x, y): return x**2 + x * y + y**2 - 3 * x for h in [(2, -1), (-3, 5)]: print(h, copy(0, [-3, 0], [[2, 1], [1, 2]], h), bowl(*h)) print(copy(2, [1, 2], [[0, 1], [1, 2]], (1, 1)), round(3 * math.e, 4))

⌘/Ctrl + Enter runs the cell

Computing H numerically

Lesson 4.1.2 estimated second partials by central differences: $f_{xx}$ from three points along one axis, and $f_{xy}$ from the four corners of a square around the point, $\frac{f_{++} - f_{+-} - f_{-+} + f_{--}}{4\delta^2}$ . Here $\delta$ is the bump, called $h$ there; in this lesson $\mathbf{h}$ is the step. Applied to every pair of variables, the corner formula fills the whole of $H$ .

As in lesson 2.8.1, a bump near $10^{-4}$ times the size of the inputs balances truncation and rounding for a second difference. The function below works for any number of variables, and is checked on $xe^y$ and on $xyz$ .

For $f(x, y, z) = xyz$ at $(1, 2, 3)$ , $\nabla f = (yz, xz, xy) = (6, 3, 2)$ , and each entry of $H$ is the third variable: $f_{xy} = z = 3$ , $f_{xz} = y = 2$ , $f_{yz} = x = 1$ . The diagonal is 0. At the step $(0.1, 0.1, 0.1)$ the copy gives 7.16, against $1.1 \times 2.1 \times 3.1 = 7.161$ .

H

import math import numpy as np def hessian(f, a, d=1e-4): a = np.array(a, dtype=float) n = len(a) E = np.eye(n) * d H = np.empty((n, n)) for i in range(n): for j in range(n): H[i, j] = (f(a + E[i] + E[j]) - f(a + E[i] - E[j]) - f(a - E[i] + E[j]) + f(a - E[i] - E[j])) / (4 * d * d) return H print(np.round(hessian(lambda p: p[0] * math.exp(p[1]), [2, 0]), 4) + 0.0) H3 = hessian(lambda p: p[0] * p[1] * p[2], [1, 2, 3]) print(np.round(H3, 4) + 0.0) h = np.array([0.1, 0.1, 0.1]) print(round(6 + np.array([6, 3, 2]) @ h + 0.5 * h @ H3 @ h, 6), round(1.1 * 2.1 * 3.1, 6))

⌘/Ctrl + Enter runs the cell

Common mistakes

The cross term has no $\frac{1}{2}$ . $\frac{1}{2}\mathbf{h}^\top H\mathbf{h} = \frac{1}{2}f_{xx}h_1^2 + f_{xy}h_1h_2 + \frac{1}{2}f_{yy}h_2^2$ . Writing $\frac{1}{2}f_{xy}h_1h_2$ halves it; writing $2f_{xy}h_1h_2$ doubles it.
Keep the $\frac{1}{2}$ in front. Without it the copy's second derivative along every direction is twice that of $f$ , as in lesson 2.7.1.
Take $\nabla f$ and $H$ at $\mathbf{a}$ , not at $\mathbf{a} + \mathbf{h}$ . They are numbers.
Use the step $\mathbf{h} = \mathbf{x} - \mathbf{a}$ , not the new point $\mathbf{x}$ .
$H$ must come out symmetric. If your $f_{xy}$ and $f_{yx}$ differ, one of them is wrong.
The cube rule holds for small steps. Far from $\mathbf{a}$ , or where the second derivatives do not exist, the copy can miss badly.

The Hessian and second-order Taylor expansion

0 of 12 correct

1.
For $f(x, y) = x^3y$ , find $f_{xx}$ at $(1, 2)$
2.
For $f(x, y) = x^3y$ , find $f_{xy}$ at $(1, 2)$
3.
For $f(x, y) = xe^{2y}$ , find $f_{yy}$ at $(1, 0)$
4.
The Hessian of a function has $-2$ in row 2, column 1. What is in row 1, column 2?
5.
For $H = \begin{pmatrix} 4 & 2 \\ 2 & 0 \end{pmatrix}$ and $\mathbf{h} = (0.1, 0.1)$ , find $\frac{1}{2}\mathbf{h}^\top H\mathbf{h}$
6.
$f(\mathbf{a}) = 5$ , $\nabla f(\mathbf{a}) = (1, -2)$ and $H = \begin{pmatrix} 2 & 1 \\ 1 & 4 \end{pmatrix}$ . Evaluate the quadratic copy at the step $\mathbf{h} = (0.2, 0.1)$
7.
In the quadratic copy, $f_{xy} = 3$ . What is the coefficient of $h_1h_2$ ?
8.
$\nabla f(\mathbf{a}) = (4, 1)$ and $H = \begin{pmatrix} 4 & 2 \\ 2 & 0 \end{pmatrix}$ . Estimate the first component of $\nabla f$ after the step $\mathbf{h} = (0, 0.5)$
9.
For $H = \begin{pmatrix} 0 & 1 \\ 1 & 2 \end{pmatrix}$ , find the second derivative of $f$ along the unit vector $\mathbf{u} = (0.6, 0.8)$
10.
Near $\mathbf{a}$ , a quadratic copy misses by 0.008 at the step $0.2\,(1, 1)$ . Estimate the miss at $0.1\,(1, 1)$
11.
Use the quadratic copy of $x^2 + xy + y^2 - 3x$ about $(0, 0)$ to evaluate it at the step $(2, 1)$
12.
For $f(x, y, z) = xyz$ , find the Hessian entry $f_{xz}$ at $(1, 2, 3)$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

H

The Hessian and second-order Taylor expansion

The Hessian

Rule: The Hessian

H moves the gradient

Rule: H moves the gradient

The second-order Taylor expansion

Rule: Second-order Taylor expansion

Example 1: $x^2y$ at $(1, 2)$

Example 2: $xe^y$ at $(2, 0)$

How fast the error falls

Rule: Error of the quadratic copy

Both sides, and a direction with no bend

The second derivative along a direction

Rule: Second derivative along u

Where the copy is exact, and where it fails

Computing H numerically

Common mistakes

The Hessian and second-order Taylor expansion

Summary card

Free further reading

The Hessian

Rule: The Hessian

H moves the gradient

Rule: H moves the gradient

The second-order Taylor expansion

Rule: Second-order Taylor expansion

Example 1: x2yx^2yx2y at (1,2)(1, 2)(1,2)

Example 2: xeyxe^yxey at (2,0)(2, 0)(2,0)

How fast the error falls

Rule: Error of the quadratic copy

Both sides, and a direction with no bend

The second derivative along a direction

Rule: Second derivative along u

Where the copy is exact, and where it fails

Computing H numerically

Common mistakes

Summary card

Free further reading

Example 1: $x^2y$ at $(1, 2)$

Example 2: $xe^y$ at $(2, 0)$