4.3.2 Minima, maxima and saddle points

In this lesson

Critical points: where $\nabla f = \mathbf{0}$ , found by solving equations
Why the Hessian decides: near a critical point the quadratic copy is $f(\mathbf{a}) + \frac{1}{2}\mathbf{h}^\top H\mathbf{h}$
The second-derivative test: $H$ positive definite gives a minimum, negative definite a maximum, indefinite a saddle; semidefinite decides nothing
The $2 \times 2$ test with $\det H$ and $f_{xx}$
Eigenvalues of $H$ as the bends of $f$ along its eigenvectors, and three variables in NumPy
Local against global, and the points the test misses

You need the Hessian and the quadratic copy from lesson 4.3.1, the gradient from lesson 4.2.1, maxima and minima in one variable from lessons 2.4.1 and 2.4.2, the tests for positive definiteness from lesson 3.7.1, and eigenvectors of symmetric matrices from lesson 3.6.3.

Critical points

In one variable, lesson 2.4.1 looked for the highest and lowest points where $f' = 0$ . In several variables, a smooth function can only have a top or a bottom where every partial derivative is 0. There the tangent plane is flat: the slope $\nabla f \cdot \mathbf{u}$ is 0 in every direction (lesson 4.2.1).

Rule: Critical point

\nabla f(\mathbf{a}) = \mathbf{0}

A point where every partial derivative is 0 is a critical point. To find them, solve the equations $f_x = 0$ , $f_y = 0$ together. A point with no gradient, such as the tip of a cone, needs a separate look.

Three kinds of critical point matter:

a local minimum: $f(\mathbf{a})$ is the lowest value of $f$ near $\mathbf{a}$ ;
a local maximum: the highest value near $\mathbf{a}$ ;
a saddle point: neither, because $f$ rises in some directions and falls in others.

Example 1: $f(x, y) = x^3 - 3x + y^2$ . Here $\nabla f = (3x^2 - 3,\ 2y)$ . The first equation gives $x = 1$ or $x = -1$ , the second $y = 0$ . So there are two critical points: $(1, 0)$ , where $f = -2$ , and $(-1, 0)$ , where $f = 2$ .

Example 2: $f(x, y) = 6xy - 2x^3 - 3y^2$ . Here $\nabla f = (6y - 6x^2,\ 6x - 6y)$ . The second equation gives $y = x$ ; then the first is $6x - 6x^2 = 0$ , so $x = 0$ or $x = 1$ . The critical points are $(0, 0)$ , where $f = 0$ , and $(1, 1)$ , where $f = 1$ .

The equations are not linear here, so there can be several solutions. For a quadratic function they are linear, and lesson 4.4.1 solves them as a system (lesson 3.3.1).

f_x = 0

import sympy as sp x, y = sp.symbols("x y", real=True) for f in [x**3 - 3 * x + y**2, 6 * x * y - 2 * x**3 - 3 * y**2]: points = sp.solve([sp.diff(f, x), sp.diff(f, y)], [x, y], dict=True) print([((p[x], p[y]), f.subs(p)) for p in points])

⌘/Ctrl + Enter runs the cell

What the Hessian says

At a critical point the gradient term of the quadratic copy (lesson 4.3.1) is 0. What is left is the quadratic form of lesson 3.7.1:

f(\mathbf{a} + \mathbf{h}) - f(\mathbf{a}) \approx \tfrac{1}{2}\,\mathbf{h}^\top H\,\mathbf{h}

So near $\mathbf{a}$ , the sign of the change in $f$ is the sign of $\mathbf{h}^\top H\mathbf{h}$ .

For the cubic, $f_{xx} = 6x$ , $f_{xy} = 0$ and $f_{yy} = 2$ , so $H = \begin{pmatrix} 6x & 0 \\ 0 & 2 \end{pmatrix}$ .

At $(1, 0)$ , $H = \begin{pmatrix} 6 & 0 \\ 0 & 2 \end{pmatrix}$ and the change is about $3h_1^2 + h_2^2$ : positive for every step. A minimum.
At $(-1, 0)$ , $H = \begin{pmatrix} -6 & 0 \\ 0 & 2 \end{pmatrix}$ and the change is about $-3h_1^2 + h_2^2$ : negative along $x$ , positive along $y$ . A saddle.

Near the saddle, here is the true change $f(-1 + h_1, h_2) - 2$ against $-3h_1^2 + h_2^2$ :

step $\mathbf{h}$	$(0.1, 0)$	$(-0.1, 0)$	$(0, 0.1)$	$(0.1, 0.1)$	$(0.1, 0.2)$
change in $f$	$-0.029$	$-0.031$	0.01	$-0.019$	0.011
$\frac{1}{2}\mathbf{h}^\top H\mathbf{h}$	$-0.03$	$-0.03$	0.01	$-0.02$	0.01

The quadratic form is 0 on the lines $h_2 = \pm\sqrt{3}\,h_1$ , which separate the steps that go up from those that go down, like the hyperbolas of lesson 3.7.1.

Slices of $x^3 - 3x + y^2$ through its two critical points. At $(1, 0)$ both slices have a bottom: a minimum. At $(-1, 0)$ the slice along $x$ has a top and the slice along $y$ a bottom: a saddle.

Lesson 3.7.1's tests now apply to the matrix of second derivatives.

Rule: The second-derivative test

\begin{gathered} \nabla f(\mathbf{a}) = \mathbf{0} \text{ and } H \text{ at } \mathbf{a}: \\[4pt] \text{positive definite} \implies \text{local minimum} \\ \text{negative definite} \implies \text{local maximum} \\ \text{indefinite} \implies \text{saddle point} \end{gathered}

If $H$ is only semidefinite, with an eigenvalue 0, the test cannot decide: the quadratic copy is flat in some direction, and the terms it leaves out take over.

Try it below. Jump to each critical point of the cubic and read the Hessian and the verdict. Then move the point off $(1, 0)$ with the sliders: $H$ still curves up every way while $x > 0$ , but the gradient is no longer 0, so the point is not a minimum. Switch to the saddle $x^2 - y^2$ to compare.

Explore

Move the point and read the Hessian, the table of second derivatives. Jump to a point where the gradient is zero and the Hessian says what kind of point it is.

Point x = 0

Point y = 1

Point: (0, 1), f = 1
Gradient ∇f (dark arrow): (−3, 2), length 3.606
Hessian: fxx = 0, fxy = 0, fyy = 2; det H = 0
So: The gradient is not 0, so this is not a critical point. det H = 0: the test cannot decide

f

def f(x, y): return x**3 - 3 * x + y**2 steps = [(0.1, 0), (-0.1, 0), (0, 0.1), (0.1, 0.1), (0.1, 0.2)] for h1, h2 in steps: print((h1, h2), round(f(-1 + h1, h2) - f(-1, 0), 6), round(-3 * h1**2 + h2**2, 6)) for h1, h2 in steps: print((h1, h2), round(f(1 + h1, h2) - f(1, 0), 6), round(3 * h1**2 + h2**2, 6))

⌘/Ctrl + Enter runs the cell

The $2 \times 2$ test: $\det H$ and $f_{xx}$

For two variables, lesson 3.7.1's leading minors are $f_{xx}$ and $\det H = f_{xx}f_{yy} - f_{xy}^2$ . A positive determinant means the two eigenvalues of $H$ have the same sign, and $f_{xx}$ says which.

Rule: Second-derivative test for two variables

\begin{gathered} \det H = f_{xx}f_{yy} - f_{xy}^2 \\[6pt] \det H > 0,\ f_{xx} > 0:\ \text{minimum} \\ \det H > 0,\ f_{xx} < 0:\ \text{maximum} \\ \det H < 0:\ \text{saddle} \\ \det H = 0:\ \text{no verdict} \end{gathered}

All evaluated at a critical point. With $\det H > 0$ , $f_{xx}$ and $f_{yy}$ have the same sign, so either one can be used.

Example 2 again. For $6xy - 2x^3 - 3y^2$ , $f_{xx} = -12x$ , $f_{xy} = 6$ and $f_{yy} = -6$ :

\begin{gathered} (1, 1):\ H = \begin{pmatrix} -12 & 6 \\ 6 & -6 \end{pmatrix},\ \det H = 36 \\[6pt] (0, 0):\ H = \begin{pmatrix} 0 & 6 \\ 6 & -6 \end{pmatrix},\ \det H = -36 \end{gathered}

At $(1, 1)$ , $\det H > 0$ and $f_{xx} = -12 < 0$ : a local maximum. At $(0, 0)$ , $\det H < 0$ : a saddle. There $f_{xx} = 0$ on its own says nothing; the determinant decides.

Algebra confirms the maximum. Expanding about $(1, 1)$ :

\begin{aligned} f(1 + h_1, 1 + h_2) ={}& 1 - 6h_1^2 + 6h_1h_2 \\ &- 3h_2^2 - 2h_1^3 \end{aligned}

There are no terms of degree 1: it is a critical point. The quadratic part is $\frac{1}{2}\mathbf{h}^\top H\mathbf{h}$ , and completing the square as in lesson 3.7.1 gives $-3h_1^2 - 3(h_1 - h_2)^2$ . That is negative for every step except $\mathbf{0}$ .

step $\mathbf{h}$	$(0.1, 0)$	$(0, 0.1)$	$(0.1, 0.1)$	$(0.1, -0.1)$	$(-0.1, 0.2)$
$f - 1$	$-0.062$	$-0.03$	$-0.032$	$-0.152$	$-0.298$
$\frac{1}{2}\mathbf{h}^\top H\mathbf{h}$	$-0.06$	$-0.03$	$-0.03$	$-0.15$	$-0.3$

The gaps are the cubic term $-2h_1^3$ : $-0.002$ for $h_1 = 0.1$ , $+0.002$ for $h_1 = -0.1$ , and 0 when $h_1 = 0$ .

A positive diagonal is not enough. $x^2 + 4xy + y^2$ has a critical point at $(0, 0)$ with $f_{xx} = f_{yy} = 2$ , but $f_{xy} = 4$ and $\det H = 4 - 16 = -12$ . It is a saddle: $f(0.1, -0.1) = -0.02$ . This is lesson 3.7.1's matrix $N$ , doubled.

2 \times 2

def kind(fxx, fxy, fyy, tol=1e-12): det = fxx * fyy - fxy**2 if abs(det) < tol: return det, "no verdict" if det < 0: return det, "saddle" return det, "minimum" if fxx > 0 else "maximum" print(kind(6, 0, 2), kind(-6, 0, 2)) # the cubic at (1, 0) and (-1, 0) print(kind(-12, 6, -6), kind(0, 6, -6)) # example 2 at (1, 1) and (0, 0) print(kind(2, 4, 2), kind(0, 0, 2)) # x^2 + 4xy + y^2, and a zero eigenvalue print(round(0.1**2 + 4 * 0.1 * (-0.1) + (-0.1)**2, 6)) def f(x, y): return 6 * x * y - 2 * x**3 - 3 * y**2 for h1, h2 in [(0.1, 0), (0, 0.1), (0.1, 0.1), (0.1, -0.1), (-0.1, 0.2)]: print((h1, h2), round(f(1 + h1, 1 + h2) - 1, 6), round(-6 * h1**2 + 6 * h1 * h2 - 3 * h2**2, 6))

⌘/Ctrl + Enter runs the cell

Eigenvalues: the bends along the eigenvectors

Lesson 4.3.1 showed that $\mathbf{u}^\top H\mathbf{u}$ is the second derivative of $f$ along a unit vector $\mathbf{u}$ . Lesson 3.6.3 wrote a symmetric matrix as $H = Q\Lambda Q^\top$ , with perpendicular unit eigenvectors $\mathbf{q}_i$ . Along an eigenvector, $\mathbf{q}_i^\top H\mathbf{q}_i = \lambda_i$ .

Rule: Eigenvalues of H as bends

\begin{gathered} \mathbf{q}_i^\top H\,\mathbf{q}_i = \lambda_i \\[4pt] f(\mathbf{a} + t\mathbf{q}_i) \approx f(\mathbf{a}) + \tfrac{1}{2}\lambda_i t^2 \end{gathered}

At a critical point, each eigenvalue is the bend of $f$ along its eigenvector. Over all unit vectors the bend runs from the smallest eigenvalue to the largest (lesson 3.7.1): a saddle has directions of both signs.

For the cubic at $(-1, 0)$ , $H$ is diagonal, so the eigenvectors are the axes: $\lambda = -6$ along $x$ , the way down, and $\lambda = 2$ along $y$ , the way up.

For example 2 at the saddle $(0, 0)$ , the eigenvalues are $-9.7082$ and $3.7082$ , with unit eigenvectors $(-0.5257, 0.8507)$ and $(0.8507, 0.5257)$ . A step of $t = 0.1$ along the first gives $f = -0.04825$ , against $\frac{1}{2}\lambda t^2 = -0.048541$ . Along the second, $f = 0.01731$ against 0.018541.

Over all unit vectors, $\mathbf{u}^\top H\mathbf{u}$ runs from $-9.708$ to $3.708$ . The bend is most negative along the first eigenvector, so near the saddle $f$ falls fastest that way.

\mathbf{u}^\top H\mathbf{u}

import numpy as np def f(x, y): return 6 * x * y - 2 * x**3 - 3 * y**2 H = np.array([[0.0, 6.0], [6.0, -6.0]]) # example 2 at (0, 0) lam, Q = np.linalg.eigh(H) for i in range(2): q = Q[:, i] * np.sign(Q[np.abs(Q[:, i]).argmax(), i]) print(round(lam[i], 4), np.round(q, 4), round(f(*(0.1 * q)), 6), round(0.5 * lam[i] * 0.1**2, 6)) t = np.linspace(0, 2 * np.pi, 721) U = np.vstack([np.cos(t), np.sin(t)]) bends = np.sum(U * (H @ U), axis=0) print(round(bends.min(), 3), round(bends.max(), 3))

⌘/Ctrl + Enter runs the cell

Three or more variables

With $n$ variables, $H$ is $n \times n$ and the test is the same: the signs of its eigenvalues, or the leading minors of lesson 3.7.1. In code, use eigvalsh and treat tiny eigenvalues as 0.

Take $f(x, y, z) = x^2 + y^2 + z^2 - xyz$ . Its gradient $(2x - yz,\ 2y - xz,\ 2z - xy)$ is $\mathbf{0}$ at the origin and at $(2, 2, 2)$ , among other points. At the origin $H = 2I$ , with eigenvalues 2, 2 and 2: a minimum, $f = 0$ .

At $(2, 2, 2)$ , $H$ has 2 on the diagonal and $-2$ everywhere else. Its eigenvalues are $-2$ , 4 and 4: a saddle, with $f = 4$ . Along the eigenvector $(1, 1, 1)$ the function falls: $f(2.1, 2.1, 2.1) = 3.969$ .

The leading minors there are 2, 0 and $-32$ . They are not all positive, so this is no minimum, and they do not alternate from a negative start, so it is no maximum. The eigenvalues say which directions go down.

x^2 + y^2 + z^2 - xyz

import numpy as np def f(x, y, z): return x**2 + y**2 + z**2 - x * y * z def grad(x, y, z): return [2 * x - y * z, 2 * y - x * z, 2 * z - x * y] def hessian3(x, y, z): return np.array([[2, -z, -y], [-z, 2, -x], [-y, -x, 2]], dtype=float) print(grad(0, 0, 0), grad(2, 2, 2)) for p in [(0, 0, 0), (2, 2, 2)]: H = hessian3(*p) minors = [round(float(np.linalg.det(H[:k, :k])), 6) + 0.0 for k in (1, 2, 3)] print(p, f(*p), np.round(np.linalg.eigvalsh(H), 6) + 0.0, minors) print(round(f(2.1, 2.1, 2.1), 6))

⌘/Ctrl + Enter runs the cell

When the test cannot decide

If $\det H = 0$ , or $H$ has an eigenvalue 0, the copy is flat along that eigenvector and the test is silent. The three functions below all have $\nabla f = \mathbf{0}$ and $H = \begin{pmatrix} 0 & 0 \\ 0 & 2 \end{pmatrix}$ at the origin.

function	$f(0.1, 0)$	$f(-0.1, 0)$	$f(0, 0.1)$	the origin is
$x^4 + y^2$	0.0001	0.0001	0.01	a minimum
$-x^4 + y^2$	$-0.0001$	$-0.0001$	0.01	a saddle
$x^3 + y^2$	0.001	$-0.001$	0.01	a saddle

Along $y$ each one curves up like $y^2$ . Along $x$ the Hessian sees nothing, and the left-out term, $x^4$ , $-x^4$ or $x^3$ , decides. Lesson 2.4.2 met the same silence in one variable with $x^4$ ; look at $f$ itself near the point, or at its higher terms (lesson 2.7.1).

Try it The loop evaluates each function at the three points of the table.

functions = { "x^4 + y^2": lambda x, y: x**4 + y**2, "-x^4 + y^2": lambda x, y: -x**4 + y**2, "x^3 + y^2": lambda x, y: x**3 + y**2, } for name, g in functions.items(): print(name, [round(g(*p), 6) for p in [(0.1, 0), (-0.1, 0), (0, 0.1)]])

⌘/Ctrl + Enter runs the cell

Local is not global

The test looks only near the point. The cubic's minimum at $(1, 0)$ has $f = -2$ , yet $f(-3, 0) = -18$ : the slice $x^3 - 3x$ falls without limit as $x$ decreases, so the cubic has no lowest point at all. Example 2's maximum, $f(1, 1) = 1$ , is beaten by $f(-2, 0) = 16$ .

To find the lowest value overall, compare the critical points with the values far out and on any edge of the domain, as lesson 2.4.1 did on an interval. Lesson 4.4.1 returns to local and global minima, and lesson 4.5.3 shows when a local minimum must be the lowest point of all.

Common mistakes

Check $\nabla f = \mathbf{0}$ first. At $(1, 1)$ the cubic has $H = \begin{pmatrix} 6 & 0 \\ 0 & 2 \end{pmatrix}$ , positive definite, but $\nabla f = (0, 2)$ : $f$ still falls towards $y = 0$ , so it is no minimum.
$f_{xx} > 0$ and $f_{yy} > 0$ do not make a minimum. $x^2 + 4xy + y^2$ has both and is a saddle. Check $\det H$ .
$\det H > 0$ alone does not make a minimum. $-x^2 - 2y^2$ has $\det H = 8$ and a maximum. Check the sign of $f_{xx}$ too.
$\det H = 0$ gives no verdict, not a saddle: $x^4 + y^2$ has a minimum and $-x^4 + y^2$ a saddle.
Local is not global. A local minimum can be far above the lowest value, and there may be no lowest value.
$\nabla f = \mathbf{0}$ misses points with no gradient. The cone $\sqrt{x^2 + y^2}$ has its minimum at its tip, where the gradient does not exist (lesson 4.2.1).

Minima, maxima and saddle points

0 of 12 correct

1.
Find the critical point of $x^2 - 4x + y^2 + 6y$ . What is its $y$ -coordinate?
2.
$x^3 - 12x + y^2$ has critical points at $(2, 0)$ and $(-2, 0)$ . Find $\det H$ at $(2, 0)$
3.
At a critical point $H = \begin{pmatrix} 4 & 1 \\ 1 & 3 \end{pmatrix}$ . Type 1 for a minimum, 2 for a maximum, 3 for a saddle
4.
At a critical point $H = \begin{pmatrix} -2 & 3 \\ 3 & -5 \end{pmatrix}$ . Type 1 for a minimum, 2 for a maximum, 3 for a saddle
5.
At a critical point $H = \begin{pmatrix} 2 & 4 \\ 4 & 2 \end{pmatrix}$ . Type 1 for a minimum, 2 for a maximum, 3 for a saddle
6.
Find the value of $6xy - 2x^3 - 3y^2$ at its local maximum
7.
At a critical point $\mathbf{a}$ , $f(\mathbf{a}) = 5$ and $H = \begin{pmatrix} 4 & 0 \\ 0 & 2 \end{pmatrix}$ . Estimate $f$ at the step $\mathbf{h} = (0.1, 0.2)$
8.
At a saddle $\mathbf{a}$ with $f(\mathbf{a}) = 2$ , $H$ has eigenvalue $-6$ along the unit eigenvector $\mathbf{q}$ . Estimate $f(\mathbf{a} + 0.1\mathbf{q})$
9.
$H$ at a critical point has eigenvalues $-1.5$ , 2 and 7. Type 1 for a minimum, 2 for a maximum, 3 for a saddle
10.
For which $k$ is the test silent for $H = \begin{pmatrix} k & 2 \\ 2 & 1 \end{pmatrix}$ ?
11.
$x^4 + y^2$ has $\det H = 0$ at the origin. Find $f(-0.1, 0)$
12.
$x^3 - 3x + y^2$ has a local minimum of $-2$ at $(1, 0)$ . Find $f(-3, 0)$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\nabla f = \mathbf{0}

Minima, maxima and saddle points

Critical points

Rule: Critical point

What the Hessian says

Rule: The second-derivative test

The $2 \times 2$ test: $\det H$ and $f_{xx}$

Rule: Second-derivative test for two variables

Eigenvalues: the bends along the eigenvectors

Rule: Eigenvalues of H as bends

Three or more variables

When the test cannot decide

Local is not global

Common mistakes

Minima, maxima and saddle points

Summary card

Free further reading

Critical points

Rule: Critical point

What the Hessian says

Rule: The second-derivative test

The 2×22 \times 22×2 test: det⁡H\det HdetH and fxxf_{xx}fxx​

Rule: Second-derivative test for two variables

Eigenvalues: the bends along the eigenvectors

Rule: Eigenvalues of H as bends

Three or more variables

When the test cannot decide

Local is not global

Common mistakes

Summary card

Free further reading

The $2 \times 2$ test: $\det H$ and $f_{xx}$