2.8.1 Finite differences and their errors

Two kinds of error

A finite difference with step $h$ goes wrong in two separate ways. The truncation error comes from stopping at a finite $h$ instead of taking the limit; it shrinks as $h$ shrinks. The rounding error comes from the computer keeping only about 16 digits; it grows as $h$ shrinks.

Both are in the units of the derivative: output units per input unit. This lesson measures each one, then picks the $h$ that keeps their sum smallest.

Truncation error, from Taylor polynomials

Lesson 2.7.1 wrote the Taylor polynomial about a point $a$ in step form: $f(a + h) \approx f(a) + f'(a)h + \frac{f''(a)}{2}h^2 + \cdots$ . Take the point $a$ to be $x$ itself, and go one term further:

\begin{aligned} f(x + h) = f(x) &+ f'(x)\,h \\ &+ \tfrac{1}{2}f''(x)\,h^2 \\ &+ \tfrac{1}{6}f'''(x)\,h^3 + \cdots \end{aligned}

A step to the left replaces $h$ by $-h$ , which flips the sign of every odd power:

\begin{aligned} f(x - h) = f(x) &- f'(x)\,h \\ &+ \tfrac{1}{2}f''(x)\,h^2 \\ &- \tfrac{1}{6}f'''(x)\,h^3 + \cdots \end{aligned}

Each finite difference of lesson 2.2.2 combines these two lines.

Forward. Subtract $f(x)$ from the first line and divide by $h$ . That leaves $f'(x) + \frac{1}{2}f''(x)\,h + \frac{1}{6}f'''(x)\,h^2 + \cdots$ .
Backward. The same with the second line leaves $f'(x) - \frac{1}{2}f''(x)\,h + \frac{1}{6}f'''(x)\,h^2 - \cdots$ .
Central. Subtract the second line from the first. The $f(x)$ and $h^2$ terms cancel, and dividing by $2h$ leaves $f'(x) + \frac{1}{6}f'''(x)\,h^2 + \cdots$ .

Rule: Truncation errors

\begin{gathered} \text{forward: } \tfrac{1}{2}f''(x)\,h \\[4pt] \text{backward: } -\tfrac{1}{2}f''(x)\,h \\[4pt] \text{central: } \tfrac{1}{6}f'''(x)\,h^2 \end{gathered}

The estimate minus the true $f'(x)$ , to leading order: the first term of the series that the formula leaves over. Forward and backward errors shrink like $h$ , the central error like $h^2$ .

For $x^3$ at 1, $f''(1) = 6$ and $f'''(1) = 6$ . The forward error is $\frac{1}{2} \cdot 6h = 3h$ and the central error $\frac{1}{6} \cdot 6h^2 = h^2$ : lesson 2.2.2's $3h$ and $h^2$ . A cubic's series stops at $h^3$ , so the forward difference is exactly $3 + 3h + h^2$ ; the extra $h^2$ is the $\frac{1}{6}f'''(x)\,h^2$ term.

A second example: $e^x$ at 1

Every derivative of $e^x$ is $e^x$ , so at 1 they all equal $e \approx 2.718282$ . The rule predicts a forward error of $\frac{e}{2}h \approx 1.359h$ and a central error of $\frac{e}{6}h^2 \approx 0.453h^2$ .

step $h$	0.1	0.01	0.001
forward error	0.14056	0.013637	0.00136
predicted, $\frac{e}{2}h$	0.135914	0.013591	0.001359
central error	$4.533 \times 10^{-3}$	$4.530 \times 10^{-5}$	$4.530 \times 10^{-7}$
predicted, $\frac{e}{6}h^2$	$4.530 \times 10^{-3}$	$4.530 \times 10^{-5}$	$4.530 \times 10^{-7}$

The predictions get better as $h$ shrinks, because the terms they leave out shrink faster.

The sign of the error

The error has the sign of the derivative in its term. Where $f'' > 0$ , the curve bends up (lesson 2.4.2): the forward difference is too large and the backward one too small.

$\ln x$ bends down, with $f''(2) = -\frac{1}{4}$ . So at 2 its forward differences fall short, by about $\frac{1}{8}h$ : $-0.012098$ , $-0.001246$ and $-0.000125$ for $h$ = 0.1, 0.01 and 0.001. Its third derivative $\frac{2}{x^3}$ is $\frac{1}{4}$ at 2, so the central difference is too large, by $4.17 \times 10^{-4}$ at $h = 0.1$ , as $\frac{1}{24}h^2$ predicts.

A zero term makes a difference better than its rule says. At 0, $\sin x$ has $f'' = 0$ , so its forward difference $\frac{\sin h}{h}$ is off by only about $-\frac{1}{6}h^2$ : 0.998334 at $h = 0.1$ , as in lesson 2.3.5. A quadratic has $f''' = 0$ everywhere, so its central difference is exact.

Where it breaks: a corner inside the step

The series needs $f$ to be smooth all the way from $x - h$ to $x + h$ . Take $|x|$ at $x = 0.001$ , where the slope is 1, with $h = 0.01$ . The points $-0.009$ and $0.011$ straddle the corner at 0, and the central difference gives 0.1.

The rule predicts no error at all, since $f''' = 0$ on each side of the corner. The estimate has measured the corner, not the slope. With $h = 0.0001$ both points stay right of the corner, and it gives 1.

Case study 2.8.4 meets the same trap at an option's strike.

e^x

import math def forward(f, x, h): return (f(x + h) - f(x)) / h def central(f, x, h): return (f(x + h) - f(x - h)) / (2 * h) e = math.e for h in [0.1, 0.01, 0.001]: fe, ce = forward(math.exp, 1, h) - e, central(math.exp, 1, h) - e print(h, round(fe, 6), round(e * h / 2, 6), f"{ce:.3e}", f"{e * h**2 / 6:.3e}") for h in [0.1, 0.01, 0.001]: print(h, round(forward(math.log, 2, h) - 0.5, 6), f"{central(math.log, 2, h) - 0.5:.2e}") print(round(central(abs, 0.001, 0.01), 6), round(central(abs, 0.001, 0.0001), 6))

⌘/Ctrl + Enter runs the cell

Rounding error

A float keeps 53 binary digits, about 16 decimal digits (lesson 2.1.1). The gap between 1 and the next float up is the machine epsilon, $\varepsilon \approx 2.2 \times 10^{-16}$ . Storing a number moves it by at most about $\frac{1}{2}\varepsilon$ of its size.

Even the step is rounded. $1 + 10^{-15}$ is stored as $1 + 5\varepsilon$ , which is $1 + 1.11 \times 10^{-15}$ . That is why lesson 2.1.1's table gave 2.22044605 instead of 2 at $h = 10^{-15}$ : the step really taken was 11% too long.

Subtracting two nearly equal numbers keeps their rounding errors but cancels their leading digits. That was the trap of lesson 2.1.5: for huge $n$ , $1 + r/n$ kept only a few digits of $r/n$ , and math.log1p avoided the subtraction. A finite difference cannot avoid it, since its top, $f(x + h) - f(x)$ , is exactly such a subtraction.

Each value of $f$ is off by up to about $\frac{1}{2}\varepsilon|f(x)|$ , so the top is off by up to about $\varepsilon|f(x)|$ . Dividing by $h$ gives the rule.

Rule: Rounding error

\begin{gathered} \text{forward: about } \frac{\varepsilon\,|f(x)|}{h} \\[4pt] \text{central: about } \frac{\varepsilon\,|f(x)|}{2h} \end{gathered}

A ceiling, not an exact size: the actual rounding error jumps around below it. It grows as $h$ shrinks: a step 10 times smaller allows a rounding error 10 times larger. A large $|f(x)|$ raises it too.

For $e^x$ at 1 the forward ceiling $\frac{\varepsilon e}{h}$ is $6.0 \times 10^{-6}$ at $h = 10^{-10}$ and $6.0 \times 10^{-4}$ at $h = 10^{-12}$ . The measured errors are $1.5 \times 10^{-6}$ and $4.3 \times 10^{-4}$ : under the ceiling, and far above the truncation errors $1.4 \times 10^{-10}$ and $1.4 \times 10^{-12}$ .

\varepsilon

import math import sys eps = sys.float_info.epsilon print(eps) print((1 + 1e-15) - 1, ((1 + 1e-15) - 1) / eps) e = math.e for h in [1e-8, 1e-10, 1e-12]: error = abs((math.exp(1 + h) - e) / h - e) print(h, f"{error:.1e}", f"{eps * e / h:.1e}", f"{e * h / 2:.1e}")

⌘/Ctrl + Enter runs the cell

The total error and the best step

The total error is at most about the truncation error plus the rounding ceiling. For the forward difference that is $\frac{1}{2}|f''(x)|\,h + \frac{\varepsilon|f(x)|}{h}$ . The first part falls as $h$ shrinks and the second rises, so the total is smallest close to where they are equal.

Rule: The best step

\begin{gathered} \text{forward: } h \approx \sqrt{\frac{2\varepsilon\,|f(x)|}{|f''(x)|}} \\[4pt] \text{central: } h \approx \left(\frac{3\varepsilon\,|f(x)|}{|f'''(x)|}\right)^{1/3} \end{gathered}

Set the truncation error equal to the rounding ceiling and solve for $h$ . When $f$ and its derivatives have about the same size, these are about $\sqrt{\varepsilon} \approx 1.5 \times 10^{-8}$ and $\varepsilon^{1/3} \approx 6 \times 10^{-6}$ .

For $e^x$ at 1 both ratios are 1, so the best steps are $\sqrt{2\varepsilon} \approx 2.1 \times 10^{-8}$ and $(3\varepsilon)^{1/3} \approx 8.7 \times 10^{-6}$ . These match lesson 2.2.2's $10^{-8}$ and $10^{-5}$ for $x^3$ at 1. At those steps the errors are $5.0 \times 10^{-8}$ (forward) and $3.7 \times 10^{-11}$ (central).

So the central difference wins twice: its error falls faster, and at its best it is about a thousand times more accurate.

step $h$	forward error	central error
$10^{-2}$	$1.4 \times 10^{-2}$	$4.5 \times 10^{-5}$
$10^{-4}$	$1.4 \times 10^{-4}$	$4.5 \times 10^{-9}$
$10^{-6}$	$1.4 \times 10^{-6}$	$1.6 \times 10^{-10}$
$10^{-8}$	$6.6 \times 10^{-9}$	$6.6 \times 10^{-9}$
$10^{-10}$	$1.5 \times 10^{-6}$	$6.7 \times 10^{-7}$

e^x

import math import sys eps, e = sys.float_info.epsilon, math.e def forward(h): return (math.exp(1 + h) - e) / h def central(h): return (math.exp(1 + h) - math.exp(1 - h)) / (2 * h) for h in [1e-2, 1e-4, 1e-6, 1e-8, 1e-10]: print(h, f"{abs(forward(h) - e):.1e}", f"{abs(central(h) - e):.1e}") hf, hc = math.sqrt(2 * eps), (3 * eps) ** (1 / 3) print(f"{hf:.1e}", f"{hc:.1e}", f"{abs(forward(hf) - e):.1e}", f"{abs(central(hc) - e):.1e}")

⌘/Ctrl + Enter runs the cell

Scale the step to the function

The best-step rule needs $f''$ or $f'''$ , the very things you are avoiding. What matters is a ratio such as $\frac{|f(x)|}{|f'''(x)|}$ , which has the units of the input cubed. Its cube root is $s$ , the scale of $f$ : roughly the distance over which $f$ changes by its own size.

Rule: A default step

\begin{gathered} \text{forward: } h = \sqrt{\varepsilon}\,s \\[4pt] \text{central: } h = \varepsilon^{1/3}\,s \end{gathered}

For powers and logarithms of $x$ , the scale is $s = |x|$ (use 1 at $x = 0$ ). For $e^x$ and $\sin x$ it is $s = 1$ , wherever $x$ is.

Two examples show both cases. Below, the relative error is the error as a fraction of the true slope.

$\sqrt{x}$ at $10^6$ . The slope is 0.0005 and $s = 10^6$ , so the central step is $\varepsilon^{1/3} \times 10^6 \approx 6.06$ . Its relative error is $8.5 \times 10^{-14}$ . The unscaled $h = 10^{-5}$ gives $1.1 \times 10^{-5}$ .
$e^x$ at 30. Here $f$ changes by its own size over a distance of about 1, whatever $x$ is. The unscaled $h = 10^{-5}$ has relative error $1.7 \times 10^{-11}$ . Scaling by $x$ gives $h = 1.8 \times 10^{-4}$ , and $5.5 \times 10^{-9}$ .

The step must also keep $x - h$ where $f$ is defined. For $\sqrt{x}$ at $10^{-6}$ , $h = 10^{-5}$ puts $x - h$ below 0, and Python raises ValueError. The scaled step, $6 \times 10^{-12}$ , gives a relative error of $2.4 \times 10^{-11}$ .

At the other extreme, a step below $\varepsilon|x|$ vanishes. $10^8 + 10^{-9}$ is stored as $10^8$ , so the forward difference of $x^2$ at $10^8$ with $h = 10^{-9}$ is 0.

10^{-5}

import math import sys def central(f, x, h): return (f(x + h) - f(x - h)) / (2 * h) def relative(estimate, true): return abs(estimate - true) / abs(true) c = sys.float_info.epsilon ** (1 / 3) print(round(c * 1e6, 2), f"{relative(central(math.sqrt, 1e6, c * 1e6), 0.0005):.1e}", f"{relative(central(math.sqrt, 1e6, 1e-5), 0.0005):.1e}") print(f"{c * 30:.1e}", f"{relative(central(math.exp, 30, 1e-5), math.exp(30)):.1e}", f"{relative(central(math.exp, 30, c * 30), math.exp(30)):.1e}") try: central(math.sqrt, 1e-6, 1e-5) except ValueError as err: print("ValueError:", err) print(f"{relative(central(math.sqrt, 1e-6, c * 1e-6), 500):.1e}") print(1e8 + 1e-9 == 1e8, ((1e8 + 1e-9)**2 - 1e8**2) / 1e-9)

⌘/Ctrl + Enter runs the cell

The second derivative

Lesson 2.4.2 estimated $f''$ by the second central difference. Add the two Taylor lines, each taken one term further, to $\frac{1}{24}f^{(4)}(x)\,h^4$ , where $f^{(4)}$ is the fourth derivative. The odd powers cancel:

\begin{aligned} &f(x + h) - 2f(x) + f(x - h) \\ &= f''(x)\,h^2 + \tfrac{1}{12}f^{(4)}(x)\,h^4 + \cdots \end{aligned}

So after dividing by $h^2$ , the truncation error is $\frac{1}{12}f^{(4)}(x)\,h^2$ . The top combines three values of $f$ , one of them doubled, so its rounding can reach about $2\varepsilon|f(x)|$ , and that is divided by $h^2$ .

Rule: Second difference: errors and best step

\begin{gathered} \text{truncation: } \tfrac{1}{12}f^{(4)}(x)\,h^2 \\[4pt] \text{rounding: about } \frac{2\varepsilon\,|f(x)|}{h^2} \\[4pt] \text{best: } h \approx \left(\frac{24\varepsilon\,|f(x)|}{|f^{(4)}(x)|}\right)^{1/4} \end{gathered}

When $f$ and $f^{(4)}$ have about the same size, the best step is about $\varepsilon^{1/4} \approx 10^{-4}$ , times the scale $s$ as before.

The best step is larger because the rounding ceiling grows 100 times for each step 10 times smaller, so it overtakes truncation sooner. For $e^x$ at 1 the best step is $(24\varepsilon)^{1/4} \approx 2.7 \times 10^{-4}$ , thirty times the central difference's.

step $h$	error	truncation $\frac{e}{12}h^2$	ceiling $\frac{2\varepsilon e}{h^2}$
$10^{-2}$	$2.3 \times 10^{-5}$	$2.3 \times 10^{-5}$	$1.2 \times 10^{-11}$
$10^{-3}$	$2.3 \times 10^{-7}$	$2.3 \times 10^{-7}$	$1.2 \times 10^{-9}$
$10^{-4}$	$3.8 \times 10^{-8}$	$2.3 \times 10^{-9}$	$1.2 \times 10^{-7}$
$10^{-5}$	$6.0 \times 10^{-6}$	$2.3 \times 10^{-11}$	$1.2 \times 10^{-5}$
$10^{-6}$	$1.2 \times 10^{-5}$	$2.3 \times 10^{-13}$	$1.2 \times 10^{-3}$

At its best step the error is $2.4 \times 10^{-8}$ , against $3.7 \times 10^{-11}$ for the first derivative: a second derivative from three values is always rougher. Lesson 2.4.2 saw the right-hand end of this table. Case study 2.4.5 measured a bond's convexity with a bump of $10^{-4}$ , close to the best step.

Case study 2.8.4 uses these steps to measure an option's sensitivities by repricing it.

The measured error of each estimate with its two parts: the truncation error from the Taylor series and the rounding ceiling. The best step sits where they cross. For the second difference the ceiling is steeper, so the crossing moves to a larger step and a larger error.

\frac{e}{12}h^2

import math import sys def second(f, x, h): return (f(x + h) - 2 * f(x) + f(x - h)) / h**2 eps, e = sys.float_info.epsilon, math.e for h in [1e-2, 1e-3, 1e-4, 1e-5, 1e-6]: print(h, f"{abs(second(math.exp, 1, h) - e):.1e}", f"{e * h**2 / 12:.1e}", f"{2 * eps * e / h**2:.1e}") best = (24 * eps) ** 0.25 print(f"{best:.1e}", f"{abs(second(math.exp, 1, best) - e):.1e}")

⌘/Ctrl + Enter runs the cell

Common mistakes

Smaller is not always better. Below the best step, rounding grows like $\frac{1}{h}$ , or like $\frac{1}{h^2}$ for the second difference.
One step does not fit every function. Scale it: about 6 for $\sqrt{x}$ at $10^6$ , but about $10^{-5}$ for $e^x$ at 30.
Use a larger step for $f''$ . About $10^{-4}$ times the scale, not $10^{-8}$ .
Keep all the points on smooth ground, inside the domain. A corner between $x - h$ and $x + h$ ruins the estimate, and $\sqrt{x}$ has no value below 0.
The error has a sign. A forward difference is too large where $f'' > 0$ and too small where $f'' < 0$ ; a backward one does the opposite.
A step below $\varepsilon|x|$ vanishes. Then $x + h$ is stored as $x$ , and the difference is 0.

Finite differences and their errors

0 of 12 correct

1.
The forward difference is off by about $\frac{1}{2}f''(x)\,h$ . Estimate its error for $e^x$ at 0 with $h = 0.02$
2.
$\ln x$ has $f''(4) = -\frac{1}{16}$ . Estimate the forward-difference error at 4 with $h = 0.1$ , sign included
3.
The central difference is off by about $\frac{1}{6}f'''(x)\,h^2$ . For $x^3 + x$ , whose $f'''$ is 6, find the error with $h = 0.05$
4.
Find the central difference of $5x^2 - 3x$ at 2 with $h = 0.5$
5.
With $\varepsilon \approx 2.2 \times 10^{-16}$ , find the forward rounding ceiling $\frac{\varepsilon|f(x)|}{h}$ when $|f(x)| = 10^4$ and $h = 10^{-10}$
6.
A forward difference has truncation error $2h$ and rounding ceiling $\frac{2 \times 10^{-16}}{h}$ . They are equal at $h = 10^k$ . What is $k$ ?
7.
A central difference has truncation error $h^2$ and rounding ceiling $\frac{10^{-15}}{h}$ . They are equal at $h = 10^k$ . What is $k$ ?
8.
The default central step is $\varepsilon^{1/3}s \approx 6 \times 10^{-6}\,s$ . What step suits $\ln x$ at $x = 5000$ ?
9.
Find the central difference of $|x|$ at $x = 0.002$ with $h = 0.01$
10.
The second difference is off by about $\frac{1}{12}f^{(4)}(x)\,h^2$ . For $x^4$ , whose fourth derivative is 24, find the error with $h = 0.1$
11.
A second difference has truncation error $h^2$ and rounding ceiling $\frac{10^{-16}}{h^2}$ . They are equal at $h = 10^k$ . What is $k$ ?
12.
For a second difference, by what factor can the rounding ceiling grow when the step is made 10 times smaller?

How to type answers

Powers with ^: x^7, 2^(1/3), a^-2.
Multiply with * or by writing side by side: 3x^2, 2(x+1).
ln(x) is the natural log, log(x, 2) is log base 2, e is Euler's number, sqrt(x) is a square root.
sin(x), cos(x) and tan(x) take radians; write pi for π and sin(x)^2 for sin²x.
Your first answer to each question is saved: it counts towards your score and brings the question back for review.
Exact answers like ln(20)/ln(3) are best. Decimals need about 4 significant figures.

\frac{1}{2}f''(x)\,h

Finite differences and their errors

Two kinds of error

Truncation error, from Taylor polynomials

Rule: Truncation errors

A second example: $e^x$ at 1

The sign of the error

Where it breaks: a corner inside the step

Rounding error

Rule: Rounding error

The total error and the best step

Rule: The best step

Scale the step to the function

Rule: A default step

The second derivative

Rule: Second difference: errors and best step

Common mistakes

Finite differences and their errors

Summary card

Free further reading

Two kinds of error

Truncation error, from Taylor polynomials

Rule: Truncation errors

A second example: exe^xex at 1

The sign of the error

Where it breaks: a corner inside the step

Rounding error

Rule: Rounding error

The total error and the best step

Rule: The best step

Scale the step to the function

Rule: A default step

The second derivative

Rule: Second difference: errors and best step

Common mistakes

Summary card

Free further reading

A second example: $e^x$ at 1