4.2.1 The gradient and directional derivatives

The gradient

Lesson 4.1.2 measured the slope of a surface $z = f(x, y)$ along the two axes. The partial derivative $f_x$ is the slope of the slice with $y$ held fixed, and $f_y$ the slope with $x$ held fixed. The gradient collects both into one vector.

Rule: Gradient

\nabla f = \begin{pmatrix} f_x \\ f_y \end{pmatrix}

Read $\nabla f$ as "grad f". It is a vector at each point $(x, y)$ , written as a column like the vectors of Level 3. Each component is in units of $f$ per unit of its own variable.

Take $f(x, y) = x^2 + xy + y^2 - 3x$ , a tilted bowl. Its partials are $f_x = 2x + y - 3$ and $f_y = x + 2y$ . At the point $(0, 1)$ the height is $f = 1$ , and

\nabla f(0, 1) = \begin{pmatrix} 0 + 1 - 3 \\ 0 + 2 \end{pmatrix} = \begin{pmatrix} -2 \\ 2 \end{pmatrix}

So, heading along the $x$ -axis from $(0, 1)$ , the surface falls 2 units per unit step. Heading along the $y$ -axis it rises 2 per unit step.

Three or more variables

A function of three variables has three partials, so its gradient has three components. For $g(x, y, z) = x^2 + yz$ at $(1, 2, 3)$ , the partials are $g_x = 2x = 2$ , $g_y = z = 3$ and $g_z = y = 2$ . So $\nabla g(1, 2, 3) = (2, 3, 2)$ .

With $n$ inputs, the gradient is a vector in $\mathbb{R}^n$ .

h

import numpy as np def gradient(f, p, h=1e-5): p = np.asarray(p, dtype=float) g = np.zeros_like(p) for i in range(len(p)): e = np.zeros_like(p) e[i] = h g[i] = (f(p + e) - f(p - e)) / (2 * h) return g def f(q): x, y = q return x**2 + x * y + y**2 - 3 * x def g(q): x, y, z = q return x**2 + y * z print(np.round(gradient(f, [0, 1]), 6)) print(np.round(gradient(g, [1, 2, 3]), 6))

⌘/Ctrl + Enter runs the cell

The slope in any direction

From a point $\mathbf{p}$ you can walk off in any direction, not only along the axes. Give the direction as a unit vector $\mathbf{u}$ (lesson 3.1.2). The points $\mathbf{p} + h\mathbf{u}$ then form a straight line through $\mathbf{p}$ , each at distance $|h|$ from it.

The height along that line is a function of one number, $h$ . Its slope at $h = 0$ is the directional derivative of $f$ at $\mathbf{p}$ in the direction $\mathbf{u}$ , written $D_{\mathbf{u}} f(\mathbf{p})$ .

For the tilted bowl at $\mathbf{p} = (0, 1)$ , take $\mathbf{u} = (0.6, 0.8)$ , a vector of length 1. Here are chord slopes (lesson 2.2.1) from both sides:

step $h$	0.1	0.01	0.001
ahead, to $\mathbf{p} + h\mathbf{u}$	0.548	0.4148	0.40148
behind, to $\mathbf{p} - h\mathbf{u}$	0.252	0.3852	0.39852

Both rows settle on 0.4. Algebra agrees: expanding gives $f(0.6h,\, 1 + 0.8h) - f(0, 1) = 0.4h + 1.48h^2$ . So the chord slope is $0.4 + 1.48h$ , which tends to 0.4.

The dot product gives the same number in one line: $\nabla f \cdot \mathbf{u} = (-2)(0.6) + (2)(0.8) = 0.4$ .

Rule: Directional derivative

\begin{gathered} D_{\mathbf{u}} f = \nabla f \cdot \mathbf{u} \\[4pt] = f_x u_1 + f_y u_2 \end{gathered}

The slope of $f$ per unit of distance in the direction of the unit vector $\mathbf{u}$ . A step $h$ along $\mathbf{u}$ moves $x$ by $hu_1$ and $y$ by $hu_2$ . Each move changes $f$ at the rate of its own partial, and the two changes add.

The partials are special cases. The direction $\mathbf{e}_1 = (1, 0)$ gives $\nabla f \cdot \mathbf{e}_1 = f_x$ , and $\mathbf{e}_2 = (0, 1)$ gives $f_y$ . In three variables the dot product has three terms: for $g$ above, along $\frac{1}{3}(1, 2, 2)$ the slope is $\frac{2 + 6 + 4}{3} = 4$ .

The direction must be a unit vector

A direction often comes as an angle or as an arrow of any length. For an angle $\alpha$ (alpha) from the positive $x$ -axis, use $\mathbf{u} = (\cos\alpha, \sin\alpha)$ , a point on the unit circle. For an arrow such as $(3, 4)$ , normalise it first (lesson 3.1.2): $\frac{1}{5}(3, 4) = (0.6, 0.8)$ .

Skipping that step is costly. $\nabla f \cdot (3, 4) = -6 + 8 = 2$ , five times the slope, because a step of $h$ along $(3, 4)$ covers a distance of $5h$ .

The slope is in units of $f$ per unit of distance. Distance in the plane only makes sense when $x$ and $y$ are measured in the same units.

\nabla f \cdot \mathbf{u}

import numpy as np def f(q): x, y = q return x**2 + x * y + y**2 - 3 * x p = np.array([0.0, 1.0]) u = np.array([0.6, 0.8]) for h in [0.1, 0.01, 0.001]: ahead = (f(p + h * u) - f(p)) / h behind = (f(p - h * u) - f(p)) / -h print(h, round(ahead, 6), round(behind, 6)) grad = np.array([-2.0, 2.0]) print(round(grad @ u, 6)) arrow = np.array([3.0, 4.0]) print(grad @ arrow, round(grad @ (arrow / np.linalg.norm(arrow)), 6))

⌘/Ctrl + Enter runs the cell

Steepest ascent and steepest descent

Since $\mathbf{u}$ has length 1, the angle rule of lesson 3.1.2 gives $\nabla f \cdot \mathbf{u} = \lVert \nabla f \rVert \cos\theta$ . Here $\theta$ (theta) is the angle between $\mathbf{u}$ and $\nabla f$ . So the slope depends only on how far $\mathbf{u}$ turns away from the gradient.

At $(0, 1)$ the gradient $(-2, 2)$ points at 135° from the $x$ -axis. Its length is $2\sqrt{2} \approx 2.8284$ . Turning $\mathbf{u}$ round the circle gives these slopes:

direction $\alpha$	0°	45°	90°	135°	180°	225°	270°	315°
slope $\nabla f \cdot \mathbf{u}$	$-2$	0	2	2.8284	2	0	$-2$	$-2.8284$

The largest slope is at 135°, along the gradient. The most negative is at 315°, straight against it. Halfway between, at 45° and 225°, the slope is 0.

Rule: Steepest slopes

\begin{gathered} -\lVert \nabla f \rVert \le \nabla f \cdot \mathbf{u} \le \lVert \nabla f \rVert \\[4pt] \text{largest along } \nabla f \\[2pt] \text{smallest along } -\nabla f \end{gathered}

The gradient points straight uphill, and its length is the steepest slope. Against it is steepest descent, with slope $-\lVert \nabla f \rVert$ , and at right angles the slope is 0. The bounds are the Cauchy–Schwarz inequality of lesson 3.1.2.

The slope at $(0, 1)$ in every direction. It peaks at 2.83, the length of $\nabla f$ , in the direction of $\nabla f$ , and is 0 along the level curve, at right angles to it.

Directions the same angle either side of the gradient have the same slope. 90° and 180° both give 2, since each is 45° from 135°.

Lesson 4.4.2 uses steepest descent to find minima: take a small step against the gradient, then repeat.

The gradient crosses level curves at right angles

A level curve (lesson 4.1.1) joins points of equal height. Walking along it, $f$ does not change, so the slope along the curve is 0. By the rule, the curve's direction is at right angles to $\nabla f$ .

At $(0, 1)$ the level curve $f = 1$ leaves at 45° and 225°, the two zeros in the table. It is a closed curve round the bottom of the bowl, and passes through $(2, 1)$ too, where $f$ is also 1.

A second example. Take the bowl $f(x, y) = x^2 + 2y^2$ at $(2, 1)$ . There $f = 6$ and $\nabla f = (2x, 4y) = (4, 4)$ . The level curve $x^2 + 2y^2 = 6$ is an ellipse, and at $(2, 1)$ it heads in the direction $(1, -1)$ : the nearby point $(2.1, 0.891628)$ is still at height 6.

The dot product $(4, 4) \cdot (1, -1) = 0$ confirms the right angle. The steepest slope is $\lVert (4, 4) \rVert = 4\sqrt{2} \approx 5.6569$ , in the direction $(0.7071, 0.7071)$ .

Uphill does not have to point straight away from the bottom. The bottom of this bowl is $(0, 0)$ , so straight away from it is the direction $(2, 1)$ ; the gradient points along $(1, 1)$ .

Gradient arrows on a grid of points of the tilted bowl, all drawn at the same scale. Each points uphill at right angles to the level curves around it. The arrows are longest where the curves crowd together, and vanish at the lowest point $(2, -1)$ .

Try it below. The dark arrow is the gradient and the dashed arrow your direction $\mathbf{u}$ .

Turn the direction until the slope peaks, and check that the peak is the gradient's length. Then turn it along the highlighted level curve, where the slope is 0, and move the point to try again.

Explore

The arrow is the gradient: it points uphill, at right angles to the level curve. Turn the direction and watch the slope in that direction: largest along the gradient, zero along the curve.

Point x = 0

Point y = 1

Direction, degrees from the x-axis = 0

Point: (0, 1), f = 1
Gradient ∇f (dark arrow): (−2, 2), length 2.828
Slope at 0°: ∇f · u = −2: downhill

(2, 1)

import math import numpy as np def f(x, y): return x**2 + 2 * y**2 for x in [2.1, 2.01]: y = math.sqrt((6 - x**2) / 2) print(x, round(y, 6), round(f(x, y), 6)) grad = np.array([4.0, 4.0]) print(grad @ np.array([1, -1])) print(round(np.linalg.norm(grad), 4), np.round(grad / np.linalg.norm(grad), 4))

⌘/Ctrl + Enter runs the cell

Where the gradient is zero, and where there is none

A zero gradient. For the tilted bowl, $2x + y - 3 = 0$ and $x + 2y = 0$ both hold at $(2, -1)$ , so $\nabla f(2, -1) = (0, 0)$ . Then $\nabla f \cdot \mathbf{u} = 0$ in every direction: the surface is flat there, with no uphill direction at all.

That point is the bottom of the bowl, where $f(2, -1) = -3$ . A flat point can also be a top or a saddle; lesson 4.3.2 tells them apart.

No gradient. The cone $f(x, y) = \sqrt{x^2 + y^2}$ gives each point's distance from the origin. Its slice along $y = 0$ is $|x|$ , which has a corner at 0 (lesson 2.2.2). So $f_x$ does not exist at the origin, and neither does the gradient.

Walking out of the origin, the cone rises with slope 1 in every direction. No vector can do that through a dot product: if $\nabla f \cdot \mathbf{u} = 1$ , then $\nabla f \cdot (-\mathbf{u}) = -1$ .

Away from the origin there is no corner. By the chain rule (lesson 2.3.3), $f_x = \frac{x}{\sqrt{x^2 + y^2}}$ and $f_y = \frac{y}{\sqrt{x^2 + y^2}}$ . At $(3, 4)$ that gives $\nabla f = (0.6, 0.8)$ , of length 1: the steepest slope of a cone is 1.

(2, -1)

import math import numpy as np def bowl(x, y): return x**2 + x * y + y**2 - 3 * x h = 1e-5 gx = (bowl(2 + h, -1) - bowl(2 - h, -1)) / (2 * h) gy = (bowl(2, -1 + h) - bowl(2, -1 - h)) / (2 * h) print(round(gx, 6) + 0.0, round(gy, 6) + 0.0) def cone(x, y): return math.sqrt(x**2 + y**2) step = 1e-3 for a in [0, 90, 180, 250]: ux, uy = math.cos(math.radians(a)), math.sin(math.radians(a)) print(a, round(cone(step * ux, step * uy) / step, 6)) print((cone(step, 0) - cone(0, 0)) / step, (cone(-step, 0) - cone(0, 0)) / -step)

⌘/Ctrl + Enter runs the cell

The gradient appears again in the rest of this module. Lesson 4.2.2 follows $f$ along a curved path with it. Case study 4.2.4 uses the same first-order slopes to explain a day's change in an option's value.

Common mistakes

Normalise the direction. Along $(3, 4)$ , the tilted bowl at $(0, 1)$ has slope 0.4, not $\nabla f \cdot (3, 4) = 2$ .
The steepest slope is the gradient's length, not the sum of its components. At $(0, 1)$ it is $2\sqrt{2}$ , not $-2 + 2 = 0$ .
Put the point into the partials. $(2x + y - 3,\; x + 2y)$ is the gradient everywhere; at $(0, 1)$ it is the vector $(-2, 2)$ .
Steepest descent is along $-\nabla f$ , with slope $-\lVert \nabla f \rVert$ : the opposite direction, not a different one.
The gradient crosses level curves; it never runs along them. Along a level curve the slope is 0.
Uphill need not point away from the bottom. On $x^2 + 2y^2$ at $(2, 1)$ the gradient is along $(1, 1)$ , not $(2, 1)$ .
No gradient at a corner. The cone rises with slope 1 out of the origin in every direction, but has no gradient there.

The gradient and directional derivatives

0 of 11 correct

1.
For $f = x^2 + 3xy$ , find the first component of $\nabla f$ at $(1, 2)$
2.
For $g = xyz$ , find the second component of $\nabla g$ at $(1, 2, 3)$
3.
Find the slope of $x^2 + y^2$ at $(1, 2)$ in the direction $\mathbf{u} = (0.6, 0.8)$
4.
Find the slope of $xy$ at $(2, 1)$ in the direction of the arrow $(3, 4)$
5.
Find the slope of $x^2 - y$ at $(1, 1)$ in the direction 60° from the $x$ -axis (type sqrt for a root)
6.
Find the steepest slope of $x^2 + 2y^2$ at $(2, 1)$ (type sqrt for a root)
7.
Find the slope of $3x + 4y$ in the direction of steepest descent
8.
At a point, $\nabla f = (2, 5)$ . For which $k$ does the direction $(k, -2)$ run along the level curve?
9.
At a point, $\nabla f$ has length 6. At what angle to $\nabla f$ , in degrees from 0 to 180, is the slope 3?
10.
$x^2 + xy + y^2 - 3x$ has $\nabla f = (0, 0)$ at one point. What is its $x$ -coordinate?
11.
Find the slope of the cone $\sqrt{x^2 + y^2}$ at $(3, 4)$ in the direction $(0, 1)$

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\nabla f = (f_x, f_y)

The gradient and directional derivatives

The gradient

Rule: Gradient

Three or more variables

The slope in any direction

Rule: Directional derivative

The direction must be a unit vector

Steepest ascent and steepest descent

Rule: Steepest slopes

The gradient crosses level curves at right angles

Where the gradient is zero, and where there is none

Common mistakes

The gradient and directional derivatives

Summary card

Free further reading