Partial derivatives, gradients and Hessians

Worked solutions

Problem 1

1. $\dfrac{\partial}{\partial x}(x^2y^3) = 2xy^3$ and $\dfrac{\partial}{\partial x}(ye^{2x}) = 2ye^{2x}$.With $y$ held fixed, $y^3$ and $y$ are constant factors, and $(e^{2x})' = 2e^{2x}$ by the chain rule.
2. $\dfrac{\partial}{\partial y}(x^2y^3) = 3x^2y^2$ and $\dfrac{\partial}{\partial y}(ye^{2x}) = e^{2x}$.Now $x$ is held fixed, so $x^2$ and $e^{2x}$ are the constant factors.
3. $\nabla f(0, 1) = (0 + 2\cdot 1\cdot e^0,\; 0 + e^0)^\top = (2, 1)^\top$.The gradient stacks the two partials as a column; evaluate each at $x = 0$, $y = 1$.
4. $\partial f/\partial x = 2xy^3 + 2ye^{2x}$, $\partial f/\partial y = 3x^2y^2 + e^{2x}$, $\nabla f(0, 1) = (2, 1)^\top$

Problem 2

1. $\nabla f = (e^y,\; xe^y + 2y)^\top$.Partial in $x$ with $y$ fixed, then in $y$ with $x$ fixed.
2. $\nabla f(1, 0) = (1, 1)^\top$.$e^0 = 1$ and $2y = 0$ at $y = 0$.
3. $\|v\| = \sqrt{3^2 + 4^2} = 5$, so $u = v/\|v\| = (\tfrac35, \tfrac45)^\top$.A rate per unit distance needs a unit vector; $v$ only names the direction.
4. $D_u f(1, 0) = \nabla f(1, 0)^\top u = \tfrac35 + \tfrac45$.The directional derivative is the gradient's inner product with the unit direction.
5. $D_u f(1, 0) = \tfrac75$, with $u = (\tfrac35, \tfrac45)^\top$Sanity check: it cannot exceed $\|\nabla f(1,0)\| = \sqrt2 \approx 1.41$ (Problem 3), and $\tfrac75 = 1.4$, because $v$ points almost along $(1, 1)^\top$.

Problem 3

1. $\nabla f = (2x, 8y)^\top$, so $\nabla f(2, 1) = (4, 8)^\top$.The two partials of a sum of one-variable terms.
2. For a unit $u$, $D_u f = \nabla f^\top u = \|\nabla f\|\cos\theta$, with $\theta$ the angle between $\nabla f$ and $u$.The inner product of two vectors is the product of their lengths and the cosine of the angle between them, and $\|u\| = 1$.
3. $D_u f$ is largest when $\theta = 0$: $u = \nabla f/\|\nabla f\| = (4, 8)^\top/\sqrt{80} = (1, 2)^\top/\sqrt5$, at rate $\|\nabla f\| = \sqrt{80} = 4\sqrt5$.$\cos\theta \le 1$, with equality only when $u$ points along the gradient.
4. The level curve through $(2, 1)$ is $x^2 + 4y^2 = 8$.$f(2, 1) = 4 + 4 = 8$, and a level curve is the set where $f$ keeps that value.
5. Let $r(t)$ be any curve on it with $r(0) = (2, 1)$. Then $f(r(t)) = 8$ for all $t$, so $\nabla f(2, 1)^\top r'(0) = 0$.Chain rule: the derivative of $f(r(t))$ is $\nabla f^\top r'(t)$, and the derivative of a constant is $0$. Every tangent to the level curve is therefore perpendicular to the gradient.
6. Concretely, $r(t) = (2\sqrt2\cos t,\; \sqrt2\sin t)$ traces the curve and passes through $(2, 1)$ at $t = \pi/4$, where $r'(\pi/4) = (-2\sqrt2\sin\tfrac\pi4,\; \sqrt2\cos\tfrac\pi4)^\top = (-2, 1)^\top$.$(2\sqrt2\cos t)^2 + 4(\sqrt2\sin t)^2 = 8$, and $\sin\tfrac\pi4 = \cos\tfrac\pi4 = 1/\sqrt2$.
7. $\nabla f(2, 1) = (4, 8)^\top$; steepest ascent along $(1, 2)^\top/\sqrt5$ at rate $4\sqrt5$; the level curve $x^2 + 4y^2 = 8$ has tangent $(-2, 1)^\top$ there, and $(4, 8)\,(-2, 1)^\top = -8 + 8 = 0$So the gradient points straight across the contours, uphill, and its length is the slope in that direction. Gradient descent steps along $-\nabla f$ for this reason.

Problem 4

1. $f_x = 3x^2y^2 - 4y$ and $f_y = 2x^3y - 4x + 3y^2$.First partials, each with the other variable held fixed.
2. $f_{xx} = \dfrac{\partial}{\partial x}(3x^2y^2 - 4y) = 6xy^2$.Differentiate $f_x$ in $x$ again; $-4y$ is constant in $x$.
3. $f_{xy} = \dfrac{\partial}{\partial y}(3x^2y^2 - 4y) = 6x^2y - 4$.Differentiate $f_x$ in $y$.
4. $f_{yx} = \dfrac{\partial}{\partial x}(2x^3y - 4x + 3y^2) = 6x^2y - 4$.Differentiate $f_y$ in $x$: a different calculation that gives the same result, because the second partials of a polynomial are continuous.
5. $f_{yy} = \dfrac{\partial}{\partial y}(2x^3y - 4x + 3y^2) = 2x^3 + 6y$.Differentiate $f_y$ in $y$.
6. $\nabla^2 f = \begin{pmatrix} 6xy^2 & 6x^2y - 4 \\ 6x^2y - 4 & 2x^3 + 6y \end{pmatrix}$, and $f_{xy} = f_{yx} = 6x^2y - 4$Row $i$ is the gradient of $\partial f/\partial x_i$ written as a row, so the Hessian is the Jacobian of $\nabla f$; symmetry means the off-diagonal entries need computing only once.

Problem 5

1. $f_x = 3x^2 + 2x + 3y$ and $f_y = 3x + 2y$.First partials.
2. $f_y = 0$ gives $y = -\tfrac32x$.Solve the linear equation first, then substitute into the other.
3. $f_x = 3x^2 + 2x - \tfrac92x = x\big(3x - \tfrac52\big) = 0$, so $x = 0$ or $x = \tfrac56$.Substitute $y = -\tfrac32 x$; a product is zero when a factor is.
4. The critical points are $(0, 0)$ and $(\tfrac56, -\tfrac54)$.$y = -\tfrac32 x$ at each root.
5. $\nabla^2 f = \begin{pmatrix} 6x + 2 & 3 \\ 3 & 2 \end{pmatrix}$.$f_{xx} = 6x + 2$, $f_{xy} = f_{yx} = 3$, $f_{yy} = 2$.
6. At $(0, 0)$: $\nabla^2 f = \begin{pmatrix} 2 & 3 \\ 3 & 2 \end{pmatrix}$ with $\det = 4 - 9 = -5$; its eigenvalues are $5$ and $-1$, along $(1, 1)^\top$ and $(1, -1)^\top$.A negative determinant is a negative product of the two eigenvalues, so they have opposite signs; $f$ curves up along one eigenvector and down along the other. Along $(1, -1)^\top$, $f(t, -t) = t^3 - t^2$, which is below $f(0,0) = 0$ for small $t \neq 0$.
7. At $(\tfrac56, -\tfrac54)$: $f_{xx} = 6\cdot\tfrac56 + 2 = 7$, and $\nabla^2 f = \begin{pmatrix} 7 & 3 \\ 3 & 2 \end{pmatrix}$ with $\det = 14 - 9 = 5$.A positive determinant means the eigenvalues share a sign, and the trace $7 + 2 = 9$, their sum, makes that sign positive: the Hessian is positive definite.
8. $(0, 0)$ is a saddle: $\det = -5 < 0$, eigenvalues $5$ and $-1$. $(\tfrac56, -\tfrac54)$ is a strict local minimum: $\det = 5 > 0$ and $f_{xx} = 7 > 0$The minimum is only local: $f(x, 0) = x^3 + x^2$ goes to $-\infty$ as $x \to -\infty$.

Problem 6

1. $\nabla(x^\top A x) = (A + A^\top)x$ and $\nabla(b^\top x) = b$.Problems 4 and 3 of the matrix-calculus page: $x$ appears on both sides of $A$, so both $A$ and $A^\top$ contribute.
2. $\nabla f = \tfrac12(A + A^\top)x - b$.The gradient is linear: scale the first term by $\tfrac12$ and subtract the second.
3. $\nabla^2 f = \tfrac12(A + A^\top)$.The Hessian is the Jacobian of $\nabla f$, and the Jacobian of $Mx$ is $M$; the constant $b$ drops out. It is symmetric, as a Hessian must be, even when $A$ is not.
4. Write $H = \tfrac12(A + A^\top)$. Then $x^\top A x = x^\top H x$.$A = H + K$ with $K = \tfrac12(A - A^\top)$, and $x^\top K x$ is a scalar equal to its own transpose $x^\top K^\top x = -x^\top K x$, so it is $0$. Only the symmetric part of $A$ is ever seen by $f$.
5. $x_1 = x_0 - H^{-1}(Hx_0 - b) = x_0 - x_0 + H^{-1}b = H^{-1}b$.Newton's step with $\nabla f(x_0) = Hx_0 - b$ and $\nabla^2 f = H$; $H^{-1}H = I$.
6. $\nabla f(x_1) = HH^{-1}b - b = 0$.So $x_1$ is the critical point, reached in one step from any $x_0$: Newton's step jumps to the critical point of the second-order Taylor model, and for a quadratic that model is $f$ itself.
7. $\nabla f = \tfrac12(A + A^\top)x - b$, $\nabla^2 f = \tfrac12(A + A^\top)$; Newton's step gives $x_1 = \big(\tfrac12(A + A^\top)\big)^{-1}b$ from any $x_0$, which is $A^{-1}b$ when $A$ is symmetricWhen $\tfrac12(A + A^\top)$ is also positive definite, $f$ is strictly convex and $x_1$ is its global minimum.

Problem 7

1. Let $Z = \sum_j e^{x_j}$, so $f = \log Z$ and $\partial Z/\partial x_i = e^{x_i}$.Only the $j = i$ term of the sum depends on $x_i$.
2. $\dfrac{\partial f}{\partial x_i} = \dfrac{1}{Z}\cdot e^{x_i} = s_i$, where $s_i = e^{x_i}/\sum_j e^{x_j}$.Chain rule: the derivative of $\log$ at $Z$ is $1/Z$, times $\partial Z/\partial x_i$. The vector $s$ is the softmax of $x$.
3. $\dfrac{\partial s_i}{\partial x_k} = \dfrac{\delta_{ik}e^{x_i}Z - e^{x_i}e^{x_k}}{Z^2} = \delta_{ik}s_i - s_is_k$, with $\delta_{ik} = 1$ if $i = k$ and $0$ otherwise.Quotient rule on $e^{x_i}/Z$: the numerator depends on $x_k$ only when $k = i$, and $\partial Z/\partial x_k = e^{x_k}$.
4. $\nabla^2 f = \operatorname{diag}(s) - ss^\top$.Entry $(i, k)$ of the Hessian is $\partial s_i/\partial x_k$; the $\delta_{ik}s_i$ terms form the diagonal matrix $\operatorname{diag}(s)$ and the $s_is_k$ terms form the outer product $ss^\top$.
5. $\nabla f = s$ and $\nabla^2 f = \operatorname{diag}(s) - ss^\top$, where $s_i = e^{x_i}/\sum_j e^{x_j}$Two checks: the Hessian is symmetric, and $\nabla^2 f\,\mathbf 1 = s - s(s^\top\mathbf 1) = 0$ because $\sum_i s_i = 1$, matching $f(x + c\mathbf 1) = f(x) + c$, which is linear along $\mathbf 1$ and so has no curvature there.

Problem 8

1. Let $g(t) = f(x + th)$; then $g'(0) = \nabla f(x)^\top h$ and $g''(0) = h^\top \nabla^2 f(x)\,h$.Chain rule: $g'(t) = \nabla f(x + th)^\top h = \sum_i h_i\,\partial f/\partial x_i$, and differentiating each $\partial f/\partial x_i$ once more along $h$ gives $\sum_{i,j} h_i h_j\,\partial^2 f/\partial x_i\,\partial x_j$.
2. $f(x + h) = g(1) \approx g(0) + g'(0) + \tfrac12 g''(0) = f(x) + \nabla f(x)^\top h + \tfrac12 h^\top \nabla^2 f(x)\,h$.The one-variable Taylor polynomial of $g$ at $0$, evaluated at $t = 1$; the error is third order in $\|h\|$.
3. At $x = 0$: $f(0) = \log n$ and $s = \tfrac1n\mathbf 1$.Every $e^{0} = 1$, so the sum is $n$ and each softmax entry is $1/n$.
4. $\nabla f(0)^\top h = \tfrac1n\sum_i h_i$.$\nabla f = s$ (Problem 7).
5. $\nabla^2 f(0) = \tfrac1n I - \tfrac1{n^2}\mathbf 1\mathbf 1^\top$, so $h^\top\nabla^2 f(0)\,h = \tfrac1n\sum_i h_i^2 - \tfrac1{n^2}\big(\sum_i h_i\big)^2$.$\nabla^2 f = \operatorname{diag}(s) - ss^\top$ (Problem 7) with $s = \tfrac1n\mathbf 1$, and $h^\top\mathbf 1 = \sum_i h_i$.
6. $\log\sum_i e^{h_i} \approx \log n + \tfrac1n\sum_i h_i + \tfrac1{2n}\sum_i h_i^2 - \tfrac1{2n^2}\big(\sum_i h_i\big)^2$Steps 2 to 5. Read with $\bar h = \tfrac1n\sum_i h_i$, it is $\log n + \bar h + \tfrac12\cdot\tfrac1n\sum_i (h_i - \bar h)^2$: the mean of the $h_i$ plus half their variance, so near $0$ log-sum-exp sits above the mean by an amount set by the spread.

Problem 9

1. $\nabla_w L = \tfrac1N\sum_n (p_n - y_n)\,x^{(n)}$.$X^\top$ has columns $x^{(n)}$, so $X^\top(p - y)$ is the sum of those columns weighted by the entries of $p - y$.
2. $\dfrac{\partial p_n}{\partial w} = p_n(1 - p_n)\,x^{(n)\top}$, a $1 \times d$ row.Chain rule: $\sigma' = \sigma(1 - \sigma)$ evaluated at $x^{(n)\top}w$, times the row derivative of $x^{(n)\top}w$, which is $x^{(n)\top}$.
3. The Jacobian of $(p_n - y_n)\,x^{(n)}$ with respect to $w$ is $x^{(n)}\,\dfrac{\partial p_n}{\partial w} = p_n(1 - p_n)\,x^{(n)}x^{(n)\top}$, a $d \times d$ matrix.$x^{(n)}$ and $y_n$ do not depend on $w$; only the scalar $p_n$ does, and a constant column times a scalar's row derivative is a column times a row.
4. $\nabla_w^2 L = \tfrac1N\sum_n p_n(1 - p_n)\,x^{(n)}x^{(n)\top} = \tfrac1N X^\top D X$.The Hessian is the Jacobian of the gradient, so sum step 3 over $n$; a weighted sum of outer products of the rows of $X$ is $X^\top \operatorname{diag}(\text{weights})X$, since $X^\top D X = \sum_n D_{nn}\,x^{(n)}x^{(n)\top}$.
5. For any $v \in \mathbb{R}^d$: $v^\top\nabla_w^2 L\,v = \tfrac1N\sum_n p_n(1 - p_n)\,(x^{(n)\top}v)^2$.$v^\top x^{(n)}x^{(n)\top}v = (x^{(n)\top}v)^2$, a square.
6. $\nabla_w^2 L = \tfrac1N X^\top D X$ with $D = \operatorname{diag}(p \odot (1 - p))$, and $v^\top \nabla_w^2 L\,v = \tfrac1N\sum_n p_n(1 - p_n)(x^{(n)\top}v)^2 \ge 0$ for every $v$, so it is positive semidefiniteEvery $p_n$ lies strictly between 0 and 1, so every weight $p_n(1 - p_n)$ is positive and every term is $\ge 0$. A PSD Hessian at every $w$ makes $L$ convex, so any critical point is a global minimum.

Problem 10

1. $f_x = 2x + cy + e^x$ and $f_y = cx + 2y$.First partials.
2. $\nabla^2 f = \begin{pmatrix} 2 + e^x & c \\ c & 2 \end{pmatrix}$.$f_{xx} = 2 + e^x$, $f_{xy} = f_{yx} = c$, $f_{yy} = 2$. It depends on $x$ but not on $y$.
3. A symmetric $2 \times 2$ matrix with positive diagonal is PSD exactly when its determinant is $\ge 0$.The determinant is the product of the eigenvalues and the trace their sum; a positive trace rules out two negative eigenvalues, so $\det \ge 0$ leaves both $\ge 0$, while $\det < 0$ forces one negative.
4. $\det\nabla^2 f = 2(2 + e^x) - c^2 = 4 + 2e^x - c^2$.Product of the diagonal minus the square of the off-diagonal entry.
5. If $c^2 \le 4$, then $\det\nabla^2 f \ge 2e^x > 0$ at every point, so the Hessian is positive definite everywhere and $f$ is convex.$4 - c^2 \ge 0$, and $e^x > 0$ for every $x$.
6. If $c^2 > 4$, then $\det\nabla^2 f < 0$ wherever $e^x < \tfrac12(c^2 - 4)$, so the Hessian has a negative eigenvalue there and $f$ is not convex.$e^x$ takes every positive value, so such $x$ always exist; at those points $f$ curves down along an eigenvector, which a convex function never does.
7. $\nabla^2 f = \begin{pmatrix} 2 + e^x & c \\ c & 2 \end{pmatrix}$, and $f$ is convex exactly when $\lvert c\rvert \le 2$The boundary case $\lvert c\rvert = 2$ is still convex because $e^x$ keeps the determinant positive; without the $e^x$ term, $x^2 \pm 2xy + y^2 = (x \pm y)^2$ would be only just convex, flat along one line.