Lagrange multipliers

A minimum with a constraint

Lesson 4.4.1 found the lowest point of a function $f(x, y)$ where its gradient is $\mathbf{0}$. Often the point may not move freely. It must satisfy an equation $g(x, y) = c$, called a constraint: $g$ is a function and $c$ a fixed number.

Take the bowl $f(x, y) = x^2 + 2y^2$ and the constraint $x + y = 3$, so $g(x, y) = x + y$ and $c = 3$. The bowl's lowest point, the origin, is not on the line. The question is which point of the line gives the smallest $f$.

On a line you can substitute. Put $y = 3 - x$, and $f$ becomes a function of one variable, $h(x) = x^2 + 2(3 - x)^2 = 3x^2 - 12x + 18$. Its slope $6x - 12$ is 0 at $x = 2$ (lesson 2.4.1), so the best point is $(2, 1)$ and the smallest value is 6.

Substitution needs the constraint solved for one variable. On a circle that brings in square roots, and with many variables it gets long. Lagrange's method handles both, and it starts from a picture.

The level curve touches the constraint

The level curves of $f$ are the ellipses $x^2 + 2y^2 = k$ (lesson 4.1.1). Walk along the line from $(0, 3)$: the ellipses you cross get smaller, down to $f = 6$ at $(2, 1)$, then grow again.

At $(2, 1)$ the line does not cross its level curve: it touches it. A crossing would mean smaller ellipses on one side, and so lower values further along the line.

Gradients say the same thing. Lesson 4.2.1 showed that $\nabla f$ is at right angles to the level curve of $f$. The line is a level curve of $g$, so $\nabla g = (1, 1)$ is at right angles to the line.

Where the two curves touch, both gradients are at right angles to the same line, so they are parallel. Elsewhere $\nabla f$ has a part along the line, and a step along the line changes $f$:

• At $(1, 2)$, $\nabla f = (2, 8)$. Its slope along the unit direction $\frac{1}{\sqrt{2}}(1, -1)$ is $-4.2426$, so stepping that way lowers $f$.
• At $(2, 1)$, $\nabla f = (4, 4) = 4\nabla g$. Its slope along the line is 0.
• At $(3, 0)$, $\nabla f = (6, 0)$ and the slope is $4.2426$: $f$ falls the other way.

The explorer keeps the point on the line $x + y = c$. Set $c$ to 3 and slide the point until $\nabla f$ and $\nabla g$ are parallel: near $(2, 1)$, with $\lambda$ close to 4. Then move $c$ and find the best point again: $\lambda$ shrinks to 0 at $c = 0$ and turns negative below it.

The Lagrangian

Lagrange's condition in components is three equations for three unknowns: $f_x = \lambda g_x$, $f_y = \lambda g_y$ and $g = c$. One function collects them, the Lagrangian.

For the bowl, $L = x^2 + 2y^2 - \lambda(x + y - 3)$, and the equations are

The first two give $x = \frac{\lambda}{2}$ and $y = \frac{\lambda}{4}$. Then $\frac{3\lambda}{4} = 3$, so $\lambda = 4$ and the point is $(2, 1)$, as substitution found.

A maximum and a minimum on a circle

Now find the largest and smallest values of $f(x, y) = x + 2y$ on the circle $x^2 + y^2 = 5$. Substitution would need $y = \pm\sqrt{5 - x^2}$. Lagrange's method does not.

Here $\nabla f = (1, 2)$ and $\nabla g = (2x, 2y)$. The equations are $1 = 2\lambda x$, $2 = 2\lambda y$ and $x^2 + y^2 = 5$. The first shows $\lambda \neq 0$, so dividing the second by the first is safe and gives $y = 2x$.

Then $5x^2 = 5$, so $x = 1$ or $x = -1$. There are two candidates:

• $(1, 2)$, with $f = 5$ and $\lambda = \frac{1}{2}$: the maximum.
• $(-1, -2)$, with $f = -5$ and $\lambda = -\frac{1}{2}$: the minimum.

The condition picks out every point where a level line touches the circle, the top and the bottom alike. Comparing the values of $f$ tells which is which. In the figure, $\nabla f$ and $\nabla g$ point the same way at the maximum and opposite ways at the minimum.

Three variables: a linear system

The method works the same way in more variables. Minimise $x^2 + 2y^2 + 3z^2$ subject to $x + y + z = 1$. The gradient condition gives $2x = \lambda$, $4y = \lambda$ and $6z = \lambda$, and the constraint is a fourth equation.

All four are linear in $x$, $y$, $z$ and $\lambda$, so they form a $4 \times 4$ system (lesson 3.3.1):

Its solution is $x = \frac{6}{11}$, $y = \frac{3}{11}$, $z = \frac{2}{11}$ and $\lambda = \frac{12}{11}$, and the smallest value is $\frac{6}{11} \approx 0.545455$. Notice that $2x$, $4y$ and $6z$ all equal the same number, $\lambda$.

A quadratic function with linear constraints always gives a linear system like this. Case study 4.5.5 solves one for a portfolio.

What $\lambda$ measures

Write $f^*(c)$ for the best value on the constraint $g = c$. Then $\lambda$ is the rate at which $f^*$ changes as $c$ changes. Its units are units of $f$ per unit of $c$.

For the bowl on $x + y = c$, the same steps as before give the best point $\left(\frac{2c}{3}, \frac{c}{3}\right)$, so $f^*(c) = \frac{2c^2}{3}$ and $\lambda = \frac{4c}{3}$.

From $c = 3$ to $3.1$, $f^*$ rises by 0.4067, close to $\lambda \times 0.1 = 0.4$. The central difference $\frac{f^*(3.1) - f^*(2.9)}{0.2}$ gives 4: for a parabola the central difference is exact (lesson 2.8.1).

The sign has a meaning too:

• Positive: raising $c$ moves the line away from the origin, and the best value rises.
• Negative: at $c = -1.5$, raising $c$ moves the line towards the origin, and the best value falls.
• Zero: at $c = 0$ the line passes through the bowl's own lowest point. The constraint costs nothing, and $\nabla f = \mathbf{0}$ there.

On the circle $x^2 + y^2 = c$ the largest value of $x + 2y$ is $\sqrt{5c}$, by the same steps. At $c = 5$ its slope is $\frac{1}{2}$, the multiplier at the maximum. At $c = 5.1$ the largest value is 5.0498, a rise of 0.4975 per unit of $c$.

Where the method needs care

It finds candidates, not answers. On the circle only their values told the maximum from the minimum. On $x + y = 2$, the product $xy$ has one candidate, $(1, 1)$, and it is the largest value, not the smallest.

A best point may not exist. The bowl has no largest value on $x + y = 3$: $f(10, -7) = 198$ and $f(100, -97) = 28{,}818$. Lagrange's method still returns its one candidate, $(2, 1)$, which is the smallest value.

The constraint needs $\nabla g \neq \mathbf{0}$. Write the same line as $(x + y - 3)^2 = 0$. Now $\nabla g = (2(x + y - 3),\ 2(x + y - 3))$, which is $(0, 0)$ at every point of the line. At $(2, 1)$ the condition would read $(4, 4) = \lambda(0, 0)$, which no $\lambda$ satisfies, so the method misses the answer.

Lesson 4.5.2 allows inequalities, such as $x + y \ge 3$, and lesson 4.5.3 says when a single candidate is sure to be the answer.

Common mistakes

1. Keep the constraint as an equation. $\nabla f = \lambda\nabla g$ alone has too many unknowns; $g = c$ is the last of the $n + 1$ equations.
2. Compare the candidates. The condition holds at a maximum as well as at a minimum: on the circle, $(1, 2)$ gives 5 and $(-1, -2)$ gives $-5$.
3. Check that a best value exists. The bowl has a smallest value on a line but no largest one.
4. Do not divide by something that may be 0. On the circle, $1 = 2\lambda x$ showed $\lambda \neq 0$ before dividing.
5. Write the constraint so that $\nabla g \neq \mathbf{0}$. $(x + y - 3)^2 = 0$ is the right line with a useless gradient.
6. Mind the sign convention. Some books write $L = f + \lambda(g - c)$, which flips the sign of $\lambda$. Here $\nabla f = \lambda\nabla g$, so $\lambda = \frac{df^*}{dc}$.