Guides / Derivatives

Partial derivatives, gradients and Hessians: hold the other variables fixed, then read the whole Hessian

A function of several variables has one partial derivative for each variable: differentiate in that variable and treat every other one as a constant. Collected into a vector they make the gradient, which points uphill; the second partials make the Hessian, which says whether a flat point is a minimum, a maximum or a saddle. The mistakes come from a chain rule that forgets the constant is still there, a Hessian test that leaves out the mixed partial, and a gradient step taken uphill.

Why it works

Freezing y means walking across the surface z = f(x, y) along a line parallel to the x-axis. Along that line f is an ordinary function of one variable, and f_x is its slope. Near a point, a smooth surface looks like a plane, and a plane's slope in any direction is a weighted mix of its slopes in the x and y directions: along a unit vector u = (u₁, u₂) it is f_xu₁ + f_yu₂ = ∇f · u. That dot product is largest when u points the same way as ∇f, which is why the gradient is the uphill direction.

The Hessian f_xxf_xyf_xyf_yy measures curvature: along a unit vector u the curvature is u^THu. Its determinant D is the product of its two eigenvalues, the curvatures in the two principal directions. D < 0 means one is up and one is down, a saddle. D > 0 means both have the same sign, and f_xx tells you which.

Example 1: partial derivatives

Let f(x, y) = x³y + sin(xy). Find f_x, f_y and ∇f(1, 0).

1. f_x = 3x²y + y cos(xy)y is a constant: x³y differentiates like 3x² times a constant, and sin(xy) needs the chain rule, whose inner derivative ∂(xy)/∂x is y.
2. f_y = x³ + x cos(xy)Now x is the constant: x³y is x³ times y, and the inner derivative of xy in y is x.
3. ∇f(1, 0) = (0 + 0, 1 + 1) = (0, 2)At (1, 0): f_x has a factor of y in each term, and cos 0 = 1. The surface is flat in the x direction there and rises with slope 2 in the y direction.

Example 2: the gradient and a directional derivative

Let f(x, y) = x² + xy. At (1, 2), find the rate of change toward (1, 1) and the steepest rate.

1. ∇f = (2x + y, x), ∇f(1, 2) = (4, 1)The two partials, evaluated at the point.
2. u = 1√2(1, 1)The direction must be a unit vector, so divide (1, 1) by its length √2.
3. ∇f · u = (4 + 1)/√2 = 5√2 ≈ 3.536The rate of change of f per unit distance in that direction.
4. |∇f(1, 2)| = √(16 + 1) = √17 ≈ 4.123The steepest rate, in the direction of (4, 1). Every directional derivative at this point lies between −√17 and √17, and 3.536 does.

Example 3: critical points and the Hessian

Find and classify the critical points of f(x, y) = x³ − 3xy + y³.

1. f_x = 3x² − 3y = 0, f_y = −3x + 3y² = 0Set both partials to 0. They say y = x² and x = y².
2. x = x⁴ ⇒ x(x³ − 1) = 0 ⇒ x = 0 or x = 1Substitute y = x² into x = y². Factor rather than divide by x, so the x = 0 solution isn't lost. The critical points are (0, 0) and (1, 1).
3. H = 6x−3−36yf_xx = 6x, f_yy = 6y, and f_xy = −3 either way around.
4. (0, 0): D = 0 · 0 − (−3)² = −9 < 0A saddle: f rises along some directions and falls along others.
5. (1, 1): D = 36 − 9 = 27 > 0, f_xx = 6 > 0A local minimum, with f(1, 1) = 1 − 3 + 1 = −1. It isn't a global one: f(x, 0) = x³ goes to −∞.

Example 4: one step of gradient descent

Take one gradient descent step on f(x, y) = x² + 3y² from (1, 1) with step size 0.1.

1. ∇f = (2x, 6y), ∇f(1, 1) = (2, 6)The gradient points uphill, more steeply in y because the y term is three times as curved.
2. (1, 1) − 0.1 · (2, 6) = (0.8, 0.4)Step against the gradient: new point = old point − step size × gradient.
3. f(0.8, 0.4) = 0.64 + 0.48 = 1.12Down from f(1, 1) = 4. This one step is the update a training loop repeats for every weight of a model.

Common mistakes, and the exact line they happen on

1. Dropping the constant from the chain rule

The function from Example 1.

1. ∂/∂x of sin(xy), with y held constantRight so far.
2. ∂/∂x sin(xy) = cos(xy)This is the mistake. Holding y constant does not make it disappear: the inner function is xy, and its x-derivative is y. The term is y cos(xy). At (1, 0) the wrong version gives f_x = 1 instead of 0.

2. Leaving the mixed partial out of the test

The point (0, 0) from Example 3.

1. f_xx(0, 0) = 0, f_yy(0, 0) = 0, f_xy = −3Right so far.
2. D = f_xxf_yy = 0, so the test is inconclusiveThis is the mistake. D is f_xxf_yy − f_xy², the full determinant of the Hessian. With the mixed term, D = −9 and (0, 0) is a saddle. You can see it directly: f(t, t) = 2t³ − 3t² is below 0 for small t, while f(t, −t) = 3t² is above it.

3. Stepping with the gradient instead of against it

The step from Example 4.

1. ∇f(1, 1) = (2, 6)Right so far.
2. (1, 1) + 0.1 · (2, 6) = (1.2, 1.6)This is the mistake. The gradient points uphill, so adding it climbs: f(1.2, 1.6) = 9.12, more than double the starting 4. Descent subtracts the gradient. If a loss rises every step from the start, check this sign before anything else.

Practice

1. Find f_x and f_y for f(x, y) = x e^xy, and evaluate the gradient at (1, 0).

   Show answer

   f_x = e^xy + xy e^xy = e^xy(1 + xy) (product rule) and f_y = x²e^xy, so ∇f(1, 0) = (1, 1).

2. Find the gradient of f(x, y) = ln(x² + y²) at (1, 1).

   Show answer

   ∇f = (2x, 2y)x² + y², so ∇f(1, 1) = (1, 1), pointing straight away from the origin.

3. Find the rate of change of f(x, y) = xy² at (2, 1) in the direction of (1, 2).

   Show answer

   ∇f(2, 1) = (y², 2xy) = (1, 4) and u = (1, 2)/√5, so the rate is (1 + 8)/√5 = 9√5 ≈ 4.025.

4. Find and classify the critical point of f(x, y) = x² + xy + y² − 3x.

   Show answer

   2x + y − 3 = 0 and x + 2y = 0 give (2, −1). D = 2 · 2 − 1² = 3 > 0 and f_xx = 2 > 0, so it is a minimum, with f(2, −1) = −3.

5. Find and classify the critical points of f(x, y) = x⁴ − 2x² + y².

   Show answer

   4x³ − 4x = 0 and 2y = 0 give (0, 0) and (±1, 0). The Hessian is diagonal with entries 12x² − 4 and 2. At (0, 0), D = −8: a saddle. At (±1, 0), D = 16 and f_xx = 8: both are minima, with f = −1.