EulerPad

Practice / Matrix calculus

Matrix inverse and log-determinant derivatives

Ten problems on differentiating through a matrix inverse and a determinant: the differential of A⁻¹, the trace trick that reads a gradient off a differential, Jacobi's formula for det and log det, the chain rule through a parametrised matrix, backprop through a linear solve and through an inverse, the Sherman–Morrison and determinant lemmas for a rank-one update, log det through a Cholesky factor, the Gaussian log-likelihood gradient with respect to that factor, and log det through the eigenvalues, with worked solutions and the mistakes that put the factors in the wrong order.

Before you start

A matrix inverse and a determinant appear wherever a Gaussian does: the log-likelihood of a covariance, the marginal likelihood of a Gaussian process, the entropy of a VAE's posterior, a Kalman update. Differentiating through them by entries is hopeless, but there is a method that makes every one of these a few lines: write the differential of the expression, move everything into a trace, and read the gradient off the trace. These ten problems build that method from the differential of A−1A^{-1} and Jacobi's formula for det⁡A\det A, then use it for what a framework actually has to do, which is backpropagate through a linear solve and an inverse, and for the parametrisations that appear in practice: a rank-one update, a Cholesky factor, and the eigenvalues. The five mistakes at the end each give an answer of the right shape: the inverse differentiated like a reciprocal, a transpose lost in the trace trick, det⁡\det confused with log⁡det⁡\log\det, a Cholesky gradient that lands above the diagonal, and a chain rule through LL⊤LL^\top that counts LL once.

  • Gradients follow the conventions of the matrix-calculus page: for a scalar f(X)f(X) with X∈Rm×nX \in \mathbb{R}^{m\times n}, ∇Xf\nabla_X f has the shape of XX and (∇Xf)ij=∂f/∂Xij(\nabla_X f)_{ij} = \partial f/\partial X_{ij}.
  • The Frobenius inner product of two matrices of the same shape is ⟨A,B⟩=tr⁡(A⊤B)=∑ijAijBij\langle A, B\rangle = \operatorname{tr}(A^\top B) = \sum_{ij}A_{ij}B_{ij}. The trace is linear, tr⁡(A⊤)=tr⁡(A)\operatorname{tr}(A^\top) = \operatorname{tr}(A), and it is cyclic: tr⁡(ABC)=tr⁡(CAB)=tr⁡(BCA)\operatorname{tr}(ABC) = \operatorname{tr}(CAB) = \operatorname{tr}(BCA) whenever the products are defined. Cyclic is not commutative: tr⁡(ABC)≠tr⁡(ACB)\operatorname{tr}(ABC) \neq \operatorname{tr}(ACB) in general.
  • The differential dXdX of a matrix variable is an arbitrary small change of the same shape. For a scalar ff, the differential dfdf is linear in dXdX, and if df=⟨C,dX⟩df = \langle C, dX\rangle for every dXdX then ∇Xf=C\nabla_X f = C: this is the trace trick, and it is proved in Problem 2. Derivatives of products follow the product rule, d(AB)=(dA)B+A dBd(AB) = (dA)B + A\,dB, in that order, because matrices do not commute.
  • The derivative of ff along a direction EE is ddtf(A+tE)\tfrac{d}{dt}f(A + tE) at t=0t = 0; for a scalar ff it equals ⟨∇Af,E⟩\langle\nabla_A f, E\rangle.
  • AA is a square invertible matrix, A−⊤A^{-\top} is (A−1)⊤=(A⊤)−1(A^{-1})^\top = (A^\top)^{-1}, and log⁡det⁡A\log\det A is taken where det⁡A>0\det A > 0. II is the identity.
  • LL is a scalar loss, and the upstream gradient of an intermediate quantity is written with gg or GG: for x=A−1bx = A^{-1}b, g=∇xLg = \nabla_x L; for Y=A−1Y = A^{-1}, G=∇YLG = \nabla_Y L.
  • A Cholesky factor LL (the letter is reused, as it is everywhere) is lower triangular with positive diagonal and Σ=LL⊤\Sigma = LL^\top; its free parameters are the n(n+1)/2n(n+1)/2 entries on and below the diagonal. tril⁡(M)\operatorname{tril}(M) keeps the entries of MM on and below the diagonal and zeroes the rest.
  • The multivariate-Gaussian page showed ∇Σlog⁡det⁡Σ=Σ−1\nabla_\Sigma\log\det\Sigma = \Sigma^{-1} for symmetric Σ\Sigma and, for the Gaussian log-likelihood ℓ(Σ)=−N2log⁡det⁡Σ−12tr⁡(Σ−1S)+const\ell(\Sigma) = -\tfrac N2\log\det\Sigma - \tfrac12\operatorname{tr}(\Sigma^{-1}S) + \text{const} with scatter matrix SS, that ∇Σℓ=−N2Σ−1+12Σ−1SΣ−1\nabla_\Sigma\ell = -\tfrac N2\Sigma^{-1} + \tfrac12\Sigma^{-1}S\Sigma^{-1}. Problems 3 and 9 use these.

Builds on: Matrix calculus conventions, Jacobians and the chain rule

Problems

  1. ·

    Show that d(A−1)=−A−1 (dA) A−1d(A^{-1}) = -A^{-1}\,(dA)\,A^{-1}, and write the derivative of A−1A^{-1} along a direction EE.

  2. ··

    Prove the trace trick: if df=tr⁡(C⊤dX)df = \operatorname{tr}(C^\top dX) for every dXdX, then ∇Xf=C\nabla_X f = C. Then use it on f(X)=a⊤X−1bf(X) = a^\top X^{-1}b with XX invertible and aa, bb fixed vectors.

  3. ··

    Show that ddtdet⁡(I+tM)∣t=0=tr⁡M\tfrac{d}{dt}\det(I + tM)\big|_{t=0} = \operatorname{tr}M, and derive Jacobi's formula d(det⁡A)=det⁡(A)tr⁡(A−1dA)d(\det A) = \det(A)\operatorname{tr}(A^{-1}dA). Deduce ∇Adet⁡A\nabla_A\det A and ∇Alog⁡det⁡A\nabla_A\log\det A.

  4. ··

    Let A(θ)A(\theta) be an invertible matrix depending on a scalar θ\theta, with derivative A′(θ)A'(\theta). Show that ddθlog⁡det⁡A(θ)=tr⁡(A−1A′)\tfrac{d}{d\theta}\log\det A(\theta) = \operatorname{tr}\big(A^{-1}A'\big) and ddθA(θ)−1=−A−1A′A−1\tfrac{d}{d\theta}A(\theta)^{-1} = -A^{-1}A'A^{-1}. Evaluate both for A(θ)=B+θIA(\theta) = B + \theta I with BB symmetric positive definite and eigenvalues λi\lambda_i, and θ>0\theta > 0.

  5. ···

    A layer solves x=A−1bx = A^{-1}b (as solve(A, b), without forming A−1A^{-1}). With upstream gradient g=∇xLg = \nabla_x L, compute ∇bL\nabla_b L and ∇AL\nabla_A L, and say what the backward pass costs.

  6. ··

    A layer computes Y=A−1Y = A^{-1} explicitly. With upstream gradient G=∇YLG = \nabla_Y L, compute ∇AL\nabla_A L in terms of YY and GG.

  7. ··

    Rank-one update. For invertible AA and vectors uu, vv with 1+v⊤A−1u≠01 + v^\top A^{-1}u \neq 0, show the Sherman–Morrison formula (A+uv⊤)−1=A−1−A−1uv⊤A−11+v⊤A−1u(A + uv^\top)^{-1} = A^{-1} - \dfrac{A^{-1}uv^\top A^{-1}}{1 + v^\top A^{-1}u} and the matrix determinant lemma det⁡(A+uv⊤)=det⁡(A) (1+v⊤A−1u)\det(A + uv^\top) = \det(A)\,(1 + v^\top A^{-1}u). Then, with 1+v⊤A−1u>01 + v^\top A^{-1}u > 0 and det⁡A>0\det A > 0, compute ∇ulog⁡det⁡(A+uv⊤)\nabla_u\log\det(A + uv^\top).

  8. ···

    Let Σ=LL⊤\Sigma = LL^\top with LL a Cholesky factor. Show that log⁡det⁡Σ=2∑ilog⁡Lii\log\det\Sigma = 2\sum_i\log L_{ii}, and compute the gradient of log⁡det⁡Σ\log\det\Sigma with respect to the free entries of LL, once directly and once through ∇Σlog⁡det⁡Σ=Σ−1\nabla_\Sigma\log\det\Sigma = \Sigma^{-1}.

  9. ···

    The Gaussian log-likelihood of Before you start, ℓ(Σ)=−N2log⁡det⁡Σ−12tr⁡(Σ−1S)+const\ell(\Sigma) = -\tfrac N2\log\det\Sigma - \tfrac12\operatorname{tr}(\Sigma^{-1}S) + \text{const}, is optimised through Σ=LL⊤\Sigma = LL^\top. Starting from ∇Σℓ\nabla_\Sigma\ell, compute the gradient with respect to the free entries of LL, simplify it using M=L−1SL−⊤M = L^{-1}SL^{-\top}, and show that it vanishes at Σ=S/N\Sigma = S/N.

  10. ··

    Let AA be symmetric with distinct eigenvalues λi\lambda_i and orthonormal eigenvectors qiq_i, and let EE be symmetric. Show that along A+tEA + tE each eigenvalue has derivative dλidt=qi⊤Eqi\tfrac{d\lambda_i}{dt} = q_i^\top Eq_i at t=0t = 0, and use it to recover ddtlog⁡det⁡(A+tE)=tr⁡(A−1E)\tfrac{d}{dt}\log\det(A + tE) = \operatorname{tr}(A^{-1}E) for positive definite AA.

Answers

  1. d(A−1)=−A−1 (dA) A−1d(A^{-1}) = -A^{-1}\,(dA)\,A^{-1}; along EE, ddt(A+tE)−1∣t=0=−A−1EA−1\tfrac{d}{dt}(A + tE)^{-1}\big|_{t=0} = -A^{-1}EA^{-1}
  2. ∇Xf=C\nabla_X f = C whenever df=tr⁡(C⊤dX)df = \operatorname{tr}(C^\top dX); ∇X(a⊤X−1b)=−X−⊤ab⊤X−⊤\nabla_X(a^\top X^{-1}b) = -X^{-\top}ab^\top X^{-\top}
  3. ddtdet⁡(I+tM)∣0=tr⁡M\tfrac{d}{dt}\det(I + tM)\big|_0 = \operatorname{tr}M; d(det⁡A)=det⁡(A)tr⁡(A−1dA)d(\det A) = \det(A)\operatorname{tr}(A^{-1}dA); ∇Adet⁡A=det⁡(A)A−⊤\nabla_A\det A = \det(A)A^{-\top} and ∇Alog⁡det⁡A=A−⊤\nabla_A\log\det A = A^{-\top}
  4. ddθlog⁡det⁡A=tr⁡(A−1A′)\tfrac{d}{d\theta}\log\det A = \operatorname{tr}(A^{-1}A'), ddθA−1=−A−1A′A−1\tfrac{d}{d\theta}A^{-1} = -A^{-1}A'A^{-1}; for B+θIB + \theta I: ∑i1/(λi+θ)\sum_i 1/(\lambda_i + \theta) and −(B+θI)−2-(B + \theta I)^{-2}
  5. ∇bL=y=A−⊤g\nabla_b L = y = A^{-\top}g and ∇AL=−yx⊤=−(A−⊤g) x⊤\nabla_A L = -yx^\top = -(A^{-\top}g)\,x^\top
  6. ∇AL=−A−⊤GA−⊤=−Y⊤GY⊤\nabla_A L = -A^{-\top}GA^{-\top} = -Y^\top GY^\top
  7. (A+uv⊤)−1=A−1−A−1uv⊤A−11+v⊤A−1u(A + uv^\top)^{-1} = A^{-1} - \dfrac{A^{-1}uv^\top A^{-1}}{1 + v^\top A^{-1}u}; det⁡(A+uv⊤)=det⁡(A)(1+v⊤A−1u)\det(A + uv^\top) = \det(A)(1 + v^\top A^{-1}u); ∇ulog⁡det⁡(A+uv⊤)=A−⊤v1+v⊤A−1u\nabla_u\log\det(A + uv^\top) = \dfrac{A^{-\top}v}{1 + v^\top A^{-1}u}
  8. log⁡det⁡Σ=2∑ilog⁡Lii\log\det\Sigma = 2\sum_i\log L_{ii}; the gradient over the free entries of LL is 2/Lii2/L_{ii} on the diagonal and 00 below it, which is tril⁡(2Σ−1L)=tril⁡(2L−⊤)\operatorname{tril}(2\Sigma^{-1}L) = \operatorname{tril}(2L^{-\top})
  9. ∇Lℓ=tril⁡(2 ∇Σℓ L)=tril⁡(L−⊤(M−NI))\nabla_L\ell = \operatorname{tril}\big(2\,\nabla_\Sigma\ell\,L\big) = \operatorname{tril}\big(L^{-\top}(M - NI)\big) with M=L−1SL−⊤M = L^{-1}SL^{-\top}; it is 00 at Σ=S/N\Sigma = S/N
  10. dλidt=qi⊤Eqi\tfrac{d\lambda_i}{dt} = q_i^\top Eq_i, and ddtlog⁡det⁡(A+tE)=∑iqi⊤Eqi/λi=tr⁡(A−1E)\tfrac{d}{dt}\log\det(A + tE) = \sum_i q_i^\top Eq_i/\lambda_i = \operatorname{tr}(A^{-1}E)

Worked solutions

Problem 1

Show that d(A−1)=−A−1 (dA) A−1d(A^{-1}) = -A^{-1}\,(dA)\,A^{-1}, and write the derivative of A−1A^{-1} along a direction EE.

  1. A−1A=IA^{-1}A = I for every invertible AA.The definition of the inverse, and the identity does not change, so its differential is 00.
  2. d(A−1) A+A−1 dA=0d(A^{-1})\,A + A^{-1}\,dA = 0.Product rule on the left side, factors kept in order; dI=0dI = 0 on the right.
  3. d(A−1) A=−A−1 dAd(A^{-1})\,A = -A^{-1}\,dA.Move the second term across.
  4. d(A−1)=−A−1 (dA) A−1d(A^{-1}) = -A^{-1}\,(dA)\,A^{-1}.Multiply both sides by A−1A^{-1} on the right. It has to go on the right, where the AA is; multiplying on the left would not cancel it.
  5. d(A−1)=−A−1 (dA) A−1d(A^{-1}) = -A^{-1}\,(dA)\,A^{-1}; along EE, ddt(A+tE)−1∣t=0=−A−1EA−1\tfrac{d}{dt}(A + tE)^{-1}\big|_{t=0} = -A^{-1}EA^{-1}For a 1×11\times1 matrix this is d(1/a)=−da/a2d(1/a) = -da/a^2. The matrix version keeps the two factors of A−1A^{-1} apart with dAdA between them, and Problem 2 onwards never needs more than this line and the trace trick.

Problem 2

Prove the trace trick: if df=tr⁡(C⊤dX)df = \operatorname{tr}(C^\top dX) for every dXdX, then ∇Xf=C\nabla_X f = C. Then use it on f(X)=a⊤X−1bf(X) = a^\top X^{-1}b with XX invertible and aa, bb fixed vectors.

  1. tr⁡(C⊤dX)=∑ijCij dXij\operatorname{tr}(C^\top dX) = \sum_{ij}C_{ij}\,dX_{ij}.Entry (j,j)(j,j) of C⊤dXC^\top dX is ∑iCijdXij\sum_i C_{ij}dX_{ij}; the trace sums over jj.
  2. df=∑ij∂f∂Xij dXijdf = \sum_{ij}\dfrac{\partial f}{\partial X_{ij}}\,dX_{ij}.The differential of a scalar function of many variables is the sum of its partial derivatives times the changes in the variables.
  3. Both are linear in the entries of dXdX and agree for every dXdX, so ∂f/∂Xij=Cij\partial f/\partial X_{ij} = C_{ij}.Take dXdX with a single nonzero entry at (i,j)(i,j): each side reduces to that one coefficient. That is ∇Xf=C\nabla_X f = C.
  4. df=a⊤ d(X−1) b=−a⊤X−1 (dX) X−1bdf = a^\top\,d(X^{-1})\,b = -a^\top X^{-1}\,(dX)\,X^{-1}b.aa and bb are constant, and Problem 1 gives the differential of the inverse.
  5. =−tr⁡(a⊤X−1 (dX) X−1b)=−tr⁡(X−1ba⊤X−1 dX)= -\operatorname{tr}\big(a^\top X^{-1}\,(dX)\,X^{-1}b\big) = -\operatorname{tr}\big(X^{-1}ba^\top X^{-1}\,dX\big).A scalar is its own trace, and the cyclic property moves the vector factor X−1bX^{-1}b from the end to the front, keeping the order X−1bX^{-1}b, then a⊤X−1a^\top X^{-1}, then dXdX.
  6. C⊤=−X−1ba⊤X−1C^\top = -X^{-1}ba^\top X^{-1}, so C=−(X−1ba⊤X−1)⊤=−X−⊤ab⊤X−⊤C = -(X^{-1}ba^\top X^{-1})^\top = -X^{-\top}ab^\top X^{-\top}.Match step 5 to the form tr⁡(C⊤dX)\operatorname{tr}(C^\top dX), then transpose: the order reverses, (ba⊤)⊤=ab⊤(ba^\top)^\top = ab^\top, and (X−1)⊤=X−⊤(X^{-1})^\top = X^{-\top}.
  7. ∇Xf=C\nabla_X f = C whenever df=tr⁡(C⊤dX)df = \operatorname{tr}(C^\top dX); ∇X(a⊤X−1b)=−X−⊤ab⊤X−⊤\nabla_X(a^\top X^{-1}b) = -X^{-\top}ab^\top X^{-\top}n×nn\times n, the shape of XX. The Gaussian page's ∇Σ(v⊤Σ−1v)=−Σ−1vv⊤Σ−1\nabla_\Sigma(v^\top\Sigma^{-1}v) = -\Sigma^{-1}vv^\top\Sigma^{-1} is the case a=b=va = b = v with XX symmetric, so that the transposes disappear.

Problem 3

Show that ddtdet⁡(I+tM)∣t=0=tr⁡M\tfrac{d}{dt}\det(I + tM)\big|_{t=0} = \operatorname{tr}M, and derive Jacobi's formula d(det⁡A)=det⁡(A)tr⁡(A−1dA)d(\det A) = \det(A)\operatorname{tr}(A^{-1}dA). Deduce ∇Adet⁡A\nabla_A\det A and ∇Alog⁡det⁡A\nabla_A\log\det A.

  1. Let μ1,…,μn\mu_1, \dots, \mu_n be the eigenvalues of MM, with multiplicity and possibly complex. Then I+tMI + tM has eigenvalues 1+tμi1 + t\mu_i.If Mw=μwMw = \mu w then (I+tM)w=(1+tμ)w(I + tM)w = (1 + t\mu)w, and the characteristic polynomial of I+tMI + tM is that of MM shifted, so multiplicities carry over.
  2. det⁡(I+tM)=∏i(1+tμi)=1+t∑iμi+O(t2)\det(I + tM) = \prod_i(1 + t\mu_i) = 1 + t\sum_i\mu_i + O(t^2).The determinant is the product of the eigenvalues; expand and collect the terms with one factor of tt.
  3. ∑iμi=tr⁡M\sum_i\mu_i = \operatorname{tr}M, so ddtdet⁡(I+tM)∣t=0=tr⁡M\tfrac{d}{dt}\det(I + tM)\big|_{t=0} = \operatorname{tr}M.The trace is the sum of the eigenvalues; the derivative at 00 is the coefficient of tt.
  4. det⁡(A+tE)=det⁡(A(I+tA−1E))=det⁡(A)det⁡(I+tA−1E)\det(A + tE) = \det\big(A(I + tA^{-1}E)\big) = \det(A)\det(I + tA^{-1}E).Factor AA out on the left, then det⁡(XY)=det⁡Xdet⁡Y\det(XY) = \det X\det Y.
  5. ddtdet⁡(A+tE)∣t=0=det⁡(A)tr⁡(A−1E)\tfrac{d}{dt}\det(A + tE)\big|_{t=0} = \det(A)\operatorname{tr}(A^{-1}E).Step 3 with M=A−1EM = A^{-1}E. Since EE is any direction, this is d(det⁡A)=det⁡(A)tr⁡(A−1dA)d(\det A) = \det(A)\operatorname{tr}(A^{-1}dA).
  6. det⁡(A)tr⁡(A−1dA)=tr⁡((det⁡(A)A−⊤)⊤dA)\det(A)\operatorname{tr}(A^{-1}dA) = \operatorname{tr}\big((\det(A)A^{-\top})^\top dA\big), so ∇Adet⁡A=det⁡(A) A−⊤\nabla_A\det A = \det(A)\,A^{-\top}.Match to tr⁡(C⊤dA)\operatorname{tr}(C^\top dA): C⊤=det⁡(A)A−1C^\top = \det(A)A^{-1}, so C=det⁡(A)A−⊤C = \det(A)A^{-\top} (Problem 2).
  7. dlog⁡det⁡A=d(det⁡A)det⁡A=tr⁡(A−1dA)d\log\det A = \dfrac{d(\det A)}{\det A} = \operatorname{tr}(A^{-1}dA), so ∇Alog⁡det⁡A=A−⊤\nabla_A\log\det A = A^{-\top}.Chain rule through log⁡\log; the det⁡A\det A cancels, which is why log⁡det⁡\log\det is the quantity that gets differentiated in practice.
  8. ddtdet⁡(I+tM)∣0=tr⁡M\tfrac{d}{dt}\det(I + tM)\big|_0 = \operatorname{tr}M; d(det⁡A)=det⁡(A)tr⁡(A−1dA)d(\det A) = \det(A)\operatorname{tr}(A^{-1}dA); ∇Adet⁡A=det⁡(A)A−⊤\nabla_A\det A = \det(A)A^{-\top} and ∇Alog⁡det⁡A=A−⊤\nabla_A\log\det A = A^{-\top}For symmetric Σ\Sigma, Σ−⊤=Σ−1\Sigma^{-\top} = \Sigma^{-1}, which is the Gaussian page's result. det⁡(A)A−⊤\det(A)A^{-\top} is the transpose of the adjugate, whose (i,j)(i,j) entry is the cofactor of AijA_{ij}: differentiating the cofactor expansion along row ii gives the same thing entry by entry.

Problem 4

Let A(θ)A(\theta) be an invertible matrix depending on a scalar θ\theta, with derivative A′(θ)A'(\theta). Show that ddθlog⁡det⁡A(θ)=tr⁡(A−1A′)\tfrac{d}{d\theta}\log\det A(\theta) = \operatorname{tr}\big(A^{-1}A'\big) and ddθA(θ)−1=−A−1A′A−1\tfrac{d}{d\theta}A(\theta)^{-1} = -A^{-1}A'A^{-1}. Evaluate both for A(θ)=B+θIA(\theta) = B + \theta I with BB symmetric positive definite and eigenvalues λi\lambda_i, and θ>0\theta > 0.

  1. dA=A′(θ) dθdA = A'(\theta)\,d\theta.AA depends on the single variable θ\theta, so its differential is its derivative times dθd\theta.
  2. dlog⁡det⁡A=tr⁡(A−1dA)=tr⁡(A−1A′) dθd\log\det A = \operatorname{tr}(A^{-1}dA) = \operatorname{tr}(A^{-1}A')\,d\theta.Problem 3, step 7, with step 1 substituted; dθd\theta is a scalar and comes out of the trace.
  3. d(A−1)=−A−1(dA)A−1=−A−1A′A−1 dθd(A^{-1}) = -A^{-1}(dA)A^{-1} = -A^{-1}A'A^{-1}\,d\theta.Problem 1 with step 1 substituted.
  4. For A=B+θIA = B + \theta I: A′=IA' = I, so ddθlog⁡det⁡A=tr⁡(A−1)\tfrac{d}{d\theta}\log\det A = \operatorname{tr}(A^{-1}) and ddθA−1=−A−2\tfrac{d}{d\theta}A^{-1} = -A^{-2}.ddθ(B+θI)=I\tfrac{d}{d\theta}(B + \theta I) = I, and A−1IA−1=A−2A^{-1}IA^{-1} = A^{-2}. Here the factors do commute, because one of them is the identity.
  5. B=QΛQ⊤B = Q\Lambda Q^\top gives A=Q(Λ+θI)Q⊤A = Q(\Lambda + \theta I)Q^\top and A−1=Q(Λ+θI)−1Q⊤A^{-1} = Q(\Lambda + \theta I)^{-1}Q^\top.BB is symmetric, so it has an orthonormal eigenbasis; QQ⊤=IQQ^\top = I lets θI\theta I be written as Q(θI)Q⊤Q(\theta I)Q^\top, and the inverse of QDQ⊤Q D Q^\top with QQ orthogonal is QD−1Q⊤Q D^{-1}Q^\top.
  6. tr⁡(A−1)=tr⁡((Λ+θI)−1)=∑i1λi+θ\operatorname{tr}(A^{-1}) = \operatorname{tr}\big((\Lambda + \theta I)^{-1}\big) = \sum_i\dfrac{1}{\lambda_i + \theta}.Cyclic trace: tr⁡(QDQ⊤)=tr⁡(DQ⊤Q)=tr⁡D\operatorname{tr}(QDQ^\top) = \operatorname{tr}(DQ^\top Q) = \operatorname{tr}D, and DD is diagonal with entries 1/(λi+θ)1/(\lambda_i + \theta).
  7. ddθlog⁡det⁡A=tr⁡(A−1A′)\tfrac{d}{d\theta}\log\det A = \operatorname{tr}(A^{-1}A'), ddθA−1=−A−1A′A−1\tfrac{d}{d\theta}A^{-1} = -A^{-1}A'A^{-1}; for B+θIB + \theta I: ∑i1/(λi+θ)\sum_i 1/(\lambda_i + \theta) and −(B+θI)−2-(B + \theta I)^{-2}B+θIB + \theta I is the ridge matrix X⊤X+λIX^\top X + \lambda I with θ=λ\theta = \lambda, and the marginal likelihood of a Gaussian process has exactly this log⁡det⁡\log\det with θ\theta the noise variance (the Gaussian-process page). The sum is positive and decreasing in θ\theta: log⁡det⁡\log\det grows with the ridge, fastest when some λi\lambda_i is small.

Problem 5

A layer solves x=A−1bx = A^{-1}b (as solve(A, b), without forming A−1A^{-1}). With upstream gradient g=∇xLg = \nabla_x L, compute ∇bL\nabla_b L and ∇AL\nabla_A L, and say what the backward pass costs.

  1. dx=d(A−1) b+A−1 db=−A−1(dA)A−1b+A−1dbdx = d(A^{-1})\,b + A^{-1}\,db = -A^{-1}(dA)A^{-1}b + A^{-1}db.Product rule on A−1bA^{-1}b, then Problem 1 for the first term.
  2. =−A−1(dA) x+A−1db= -A^{-1}(dA)\,x + A^{-1}db.A−1b=xA^{-1}b = x, the forward output, which is saved.
  3. dL=g⊤dx=−g⊤A−1(dA) x+g⊤A−1dbdL = g^\top dx = -g^\top A^{-1}(dA)\,x + g^\top A^{-1}db.LL depends on AA and bb only through xx, and dL=∑i(∂L/∂xi)dxi=g⊤dxdL = \sum_i (\partial L/\partial x_i)dx_i = g^\top dx.
  4. Let y=A−⊤gy = A^{-\top}g, so g⊤A−1=y⊤g^\top A^{-1} = y^\top.y⊤=(A−⊤g)⊤=g⊤A−1y^\top = (A^{-\top}g)^\top = g^\top A^{-1}. yy is one solve with A⊤A^\top: solve(A.T, g).
  5. g⊤A−1db=y⊤dbg^\top A^{-1}db = y^\top db, so ∇bL=y=A−⊤g\nabla_b L = y = A^{-\top}g.Matching dL=⟨∇bL,db⟩dL = \langle\nabla_b L, db\rangle with dA=0dA = 0.
  6. −g⊤A−1(dA) x=−y⊤(dA) x=−tr⁡(y⊤ dA x)=−tr⁡(xy⊤ dA)-g^\top A^{-1}(dA)\,x = -y^\top(dA)\,x = -\operatorname{tr}(y^\top\,dA\,x) = -\operatorname{tr}(xy^\top\,dA).A scalar is its own trace; the cyclic property moves xx to the front.
  7. ∇AL=−(xy⊤)⊤=−yx⊤\nabla_A L = -(xy^\top)^\top = -yx^\top.Trace trick with C⊤=−xy⊤C^\top = -xy^\top.
  8. ∇bL=y=A−⊤g\nabla_b L = y = A^{-\top}g and ∇AL=−yx⊤=−(A−⊤g) x⊤\nabla_A L = -yx^\top = -(A^{-\top}g)\,x^\top∇AL\nabla_A L is n×nn\times n and rank one: an outer product of the solve with A⊤A^\top and the forward output. The backward pass costs one more solve (or one more triangular pair if the factorisation of AA is kept) and an outer product; A−1A^{-1} is never formed, forward or backward. Check against Problem 2: L=a⊤xL = a^\top x has g=ag = a, and −A−⊤a (A−1b)⊤=−A−⊤ab⊤A−⊤-A^{-\top}a\,(A^{-1}b)^\top = -A^{-\top}ab^\top A^{-\top}.

Problem 6

A layer computes Y=A−1Y = A^{-1} explicitly. With upstream gradient G=∇YLG = \nabla_Y L, compute ∇AL\nabla_A L in terms of YY and GG.

  1. dL=⟨G,dY⟩=tr⁡(G⊤dY)dL = \langle G, dY\rangle = \operatorname{tr}(G^\top dY).LL depends on AA only through YY, and the differential of a scalar function of a matrix is the Frobenius inner product of its gradient with the change.
  2. dY=−A−1(dA)A−1=−Y(dA)YdY = -A^{-1}(dA)A^{-1} = -Y(dA)Y.Problem 1, with A−1=YA^{-1} = Y saved from the forward pass.
  3. dL=−tr⁡(G⊤Y dA Y)=−tr⁡(YG⊤Y dA)dL = -\operatorname{tr}(G^\top Y\,dA\,Y) = -\operatorname{tr}(YG^\top Y\,dA).Cyclic: the trailing YY moves to the front.
  4. ∇AL=−(YG⊤Y)⊤=−Y⊤GY⊤\nabla_A L = -(YG^\top Y)^\top = -Y^\top GY^\top.Trace trick with C⊤=−YG⊤YC^\top = -YG^\top Y; transposing reverses the order and (G⊤)⊤=G(G^\top)^\top = G.
  5. ∇AL=−A−⊤GA−⊤=−Y⊤GY⊤\nabla_A L = -A^{-\top}GA^{-\top} = -Y^\top GY^\topn×nn\times n, the shape of AA. For L=⟨G,Y⟩L = \langle G, Y\rangle with G=ab⊤G = ab^\top, L=a⊤Yb=a⊤A−1bL = a^\top Yb = a^\top A^{-1}b and this gives −A−⊤ab⊤A−⊤-A^{-\top}ab^\top A^{-\top}, Problem 2 again. The two transposes are the whole content; when AA is symmetric and GG is not, −YGY-YGY is still wrong, because GG sits between them.

Problem 7

Rank-one update. For invertible AA and vectors uu, vv with 1+v⊤A−1u≠01 + v^\top A^{-1}u \neq 0, show the Sherman–Morrison formula (A+uv⊤)−1=A−1−A−1uv⊤A−11+v⊤A−1u(A + uv^\top)^{-1} = A^{-1} - \dfrac{A^{-1}uv^\top A^{-1}}{1 + v^\top A^{-1}u} and the matrix determinant lemma det⁡(A+uv⊤)=det⁡(A) (1+v⊤A−1u)\det(A + uv^\top) = \det(A)\,(1 + v^\top A^{-1}u). Then, with 1+v⊤A−1u>01 + v^\top A^{-1}u > 0 and det⁡A>0\det A > 0, compute ∇ulog⁡det⁡(A+uv⊤)\nabla_u\log\det(A + uv^\top).

Write s=1+v⊤A−1us = 1 + v^\top A^{-1}u, a scalar.

  1. (A+uv⊤)(A−1−A−1uv⊤A−1s)=I+uv⊤A−1−uv⊤A−1s−u (v⊤A−1u) v⊤A−1s(A + uv^\top)\Big(A^{-1} - \dfrac{A^{-1}uv^\top A^{-1}}{s}\Big) = I + uv^\top A^{-1} - \dfrac{uv^\top A^{-1}}{s} - \dfrac{u\,(v^\top A^{-1}u)\,v^\top A^{-1}}{s}.Multiply out the four terms; AA−1=IAA^{-1} = I, and in the last term the scalar v⊤A−1uv^\top A^{-1}u has been pulled out of the middle of uv⊤A−1uv⊤A−1uv^\top A^{-1}uv^\top A^{-1}.
  2. =I+uv⊤A−1(1−1s−v⊤A−1us)=I+uv⊤A−1⋅s−1−v⊤A−1us=I= I + uv^\top A^{-1}\Big(1 - \dfrac1s - \dfrac{v^\top A^{-1}u}{s}\Big) = I + uv^\top A^{-1}\cdot\dfrac{s - 1 - v^\top A^{-1}u}{s} = I.The last three terms share the factor uv⊤A−1uv^\top A^{-1}; the bracket's numerator is s−1−(s−1)=0s - 1 - (s - 1) = 0. A right inverse of a square matrix is its inverse.
  3. det⁡(I+wv⊤)=1+v⊤w\det(I + wv^\top) = 1 + v^\top w for any vectors ww, vv.I+wv⊤I + wv^\top sends ww to (1+v⊤w)w(1 + v^\top w)w, so ww is an eigenvector with eigenvalue 1+v⊤w1 + v^\top w (when w≠0w \neq 0); every zz with v⊤z=0v^\top z = 0, an (n−1)(n-1)-dimensional space, is an eigenvector with eigenvalue 11. When v⊤w≠0v^\top w \neq 0 these nn eigenvectors are independent and the determinant is the product of the eigenvalues; when v⊤w=0v^\top w = 0 both sides equal 11 by continuity in ww.
  4. det⁡(A+uv⊤)=det⁡(A(I+A−1uv⊤))=det⁡(A)det⁡(I+A−1u v⊤)=det⁡(A) (1+v⊤A−1u)\det(A + uv^\top) = \det\big(A(I + A^{-1}uv^\top)\big) = \det(A)\det(I + A^{-1}u\,v^\top) = \det(A)\,(1 + v^\top A^{-1}u).Factor out AA, split the determinant, and apply step 3 with w=A−1uw = A^{-1}u.
  5. log⁡det⁡(A+uv⊤)=log⁡det⁡A+log⁡(1+v⊤A−1u)\log\det(A + uv^\top) = \log\det A + \log(1 + v^\top A^{-1}u).Log of step 4; both factors are positive by assumption.
  6. ∇u(v⊤A−1u)=(v⊤A−1)⊤=A−⊤v\nabla_u(v^\top A^{-1}u) = (v^\top A^{-1})^\top = A^{-\top}v.v⊤A−1u=c⊤uv^\top A^{-1}u = c^\top u with c=(v⊤A−1)⊤c = (v^\top A^{-1})^\top, and ∇u(c⊤u)=c\nabla_u(c^\top u) = c (the matrix-calculus page).
  7. (A+uv⊤)−1=A−1−A−1uv⊤A−11+v⊤A−1u(A + uv^\top)^{-1} = A^{-1} - \dfrac{A^{-1}uv^\top A^{-1}}{1 + v^\top A^{-1}u}; det⁡(A+uv⊤)=det⁡(A)(1+v⊤A−1u)\det(A + uv^\top) = \det(A)(1 + v^\top A^{-1}u); ∇ulog⁡det⁡(A+uv⊤)=A−⊤v1+v⊤A−1u\nabla_u\log\det(A + uv^\top) = \dfrac{A^{-\top}v}{1 + v^\top A^{-1}u}The first two turn an O(n3)O(n^3) inverse and determinant into O(n2)O(n^2) work once A−1A^{-1} (or a factorisation) is known: adding one example to a kernel matrix, one observation to a Kalman covariance, or a rank-one step in a quasi-Newton method. The gradient agrees with Problem 3 through the chain rule: ∇Alog⁡det⁡\nabla_A\log\det at A+uv⊤A + uv^\top is (A+uv⊤)−⊤(A + uv^\top)^{-\top}, and uu enters through uv⊤uv^\top.

Problem 8

Let Σ=LL⊤\Sigma = LL^\top with LL a Cholesky factor. Show that log⁡det⁡Σ=2∑ilog⁡Lii\log\det\Sigma = 2\sum_i\log L_{ii}, and compute the gradient of log⁡det⁡Σ\log\det\Sigma with respect to the free entries of LL, once directly and once through ∇Σlog⁡det⁡Σ=Σ−1\nabla_\Sigma\log\det\Sigma = \Sigma^{-1}.

  1. det⁡Σ=det⁡L det⁡L⊤=(det⁡L)2=∏iLii2\det\Sigma = \det L\,\det L^\top = (\det L)^2 = \prod_i L_{ii}^2.det⁡(XY)=det⁡Xdet⁡Y\det(XY) = \det X\det Y, det⁡L⊤=det⁡L\det L^\top = \det L, and the determinant of a triangular matrix is the product of its diagonal.
  2. log⁡det⁡Σ=2∑ilog⁡Lii\log\det\Sigma = 2\sum_i\log L_{ii}.Log of step 1; each Lii>0L_{ii} > 0 so the logs are defined. This is how a library computes log⁡det⁡\log\det of a covariance: no determinant is ever formed.
  3. Directly: ∂∂Lij 2∑klog⁡Lkk={2/Liii=j0i>j\dfrac{\partial}{\partial L_{ij}}\,2\sum_k\log L_{kk} = \begin{cases}2/L_{ii} & i = j\\ 0 & i > j\end{cases}.Only the diagonal entries appear in the formula, so the below-diagonal entries have zero derivative, and ddxlog⁡x=1/x\tfrac{d}{dx}\log x = 1/x.
  4. Through Σ\Sigma: dΣ=(dL)L⊤+L(dL)⊤d\Sigma = (dL)L^\top + L(dL)^\top.Product rule on LL⊤LL^\top, in order.
  5. dlog⁡det⁡Σ=tr⁡(Σ−1dΣ)=tr⁡(Σ−1(dL)L⊤)+tr⁡(Σ−1L(dL)⊤)d\log\det\Sigma = \operatorname{tr}(\Sigma^{-1}d\Sigma) = \operatorname{tr}\big(\Sigma^{-1}(dL)L^\top\big) + \operatorname{tr}\big(\Sigma^{-1}L(dL)^\top\big).Problem 3, step 7, with dΣd\Sigma from step 4 and the trace split over the sum.
  6. tr⁡(Σ−1(dL)L⊤)=tr⁡(L⊤Σ−1dL)\operatorname{tr}\big(\Sigma^{-1}(dL)L^\top\big) = \operatorname{tr}\big(L^\top\Sigma^{-1}dL\big) and tr⁡(Σ−1L(dL)⊤)=tr⁡(dL L⊤Σ−1)=tr⁡(L⊤Σ−1dL)\operatorname{tr}\big(\Sigma^{-1}L(dL)^\top\big) = \operatorname{tr}\big(dL\,L^\top\Sigma^{-1}\big) = \operatorname{tr}\big(L^\top\Sigma^{-1}dL\big).Cyclic trace for the first; for the second, transpose inside the trace (which leaves it unchanged, using Σ−⊤=Σ−1\Sigma^{-\top} = \Sigma^{-1}) and then cycle. The two terms are equal.
  7. dlog⁡det⁡Σ=2tr⁡((Σ−1L)⊤dL)d\log\det\Sigma = 2\operatorname{tr}\big((\Sigma^{-1}L)^\top dL\big), so over all entries the gradient is 2Σ−1L=2L−⊤L−1L=2L−⊤2\Sigma^{-1}L = 2L^{-\top}L^{-1}L = 2L^{-\top}.Trace trick with C⊤=2L⊤Σ−1C^\top = 2L^\top\Sigma^{-1}, so C=2Σ−1LC = 2\Sigma^{-1}L; then Σ−1=(LL⊤)−1=L−⊤L−1\Sigma^{-1} = (LL^\top)^{-1} = L^{-\top}L^{-1}.
  8. L−⊤L^{-\top} is upper triangular with diagonal 1/Lii1/L_{ii}, so tril⁡(2L−⊤)=diag⁡(2/L11,…,2/Lnn)\operatorname{tril}(2L^{-\top}) = \operatorname{diag}(2/L_{11}, \dots, 2/L_{nn}).The inverse of a lower-triangular matrix is lower triangular with reciprocal diagonal, and its transpose is upper triangular. Only the entries where LL has free parameters count: dLdL is zero above the diagonal, so those entries of CC multiply nothing.
  9. log⁡det⁡Σ=2∑ilog⁡Lii\log\det\Sigma = 2\sum_i\log L_{ii}; the gradient over the free entries of LL is 2/Lii2/L_{ii} on the diagonal and 00 below it, which is tril⁡(2Σ−1L)=tril⁡(2L−⊤)\operatorname{tril}(2\Sigma^{-1}L) = \operatorname{tril}(2L^{-\top})Both routes agree, and the second is the one a framework's autodiff takes: it computes the full 2Σ−1L2\Sigma^{-1}L and the restriction to the parametrised entries discards the upper triangle. A gradient step on the off-diagonal entries of LL that came from log⁡det⁡\log\det alone would be a bug; those entries only move through the other terms of a likelihood (Problem 9).

Problem 9

The Gaussian log-likelihood of Before you start, ℓ(Σ)=−N2log⁡det⁡Σ−12tr⁡(Σ−1S)+const\ell(\Sigma) = -\tfrac N2\log\det\Sigma - \tfrac12\operatorname{tr}(\Sigma^{-1}S) + \text{const}, is optimised through Σ=LL⊤\Sigma = LL^\top. Starting from ∇Σℓ\nabla_\Sigma\ell, compute the gradient with respect to the free entries of LL, simplify it using M=L−1SL−⊤M = L^{-1}SL^{-\top}, and show that it vanishes at Σ=S/N\Sigma = S/N.

Write C=∇Σℓ=−N2Σ−1+12Σ−1SΣ−1C = \nabla_\Sigma\ell = -\tfrac N2\Sigma^{-1} + \tfrac12\Sigma^{-1}S\Sigma^{-1}, which is symmetric because Σ\Sigma and SS are.

  1. dℓ=⟨C,dΣ⟩=⟨C,(dL)L⊤⟩+⟨C,L(dL)⊤⟩d\ell = \langle C, d\Sigma\rangle = \langle C, (dL)L^\top\rangle + \langle C, L(dL)^\top\rangle.ℓ\ell depends on LL only through Σ\Sigma, with dΣd\Sigma from Problem 8, step 4, and the inner product is linear.
  2. ⟨C,(dL)L⊤⟩=tr⁡(C⊤(dL)L⊤)=tr⁡(L⊤C⊤dL)=⟨CL,dL⟩\langle C, (dL)L^\top\rangle = \operatorname{tr}\big(C^\top(dL)L^\top\big) = \operatorname{tr}\big(L^\top C^\top dL\big) = \langle CL, dL\rangle.Cyclic trace, then L⊤C⊤=(CL)⊤L^\top C^\top = (CL)^\top.
  3. ⟨C,L(dL)⊤⟩=tr⁡(C⊤L(dL)⊤)=tr⁡(dL L⊤C)=tr⁡(L⊤C dL)=⟨C⊤L,dL⟩\langle C, L(dL)^\top\rangle = \operatorname{tr}\big(C^\top L(dL)^\top\big) = \operatorname{tr}\big(dL\,L^\top C\big) = \operatorname{tr}\big(L^\top C\,dL\big) = \langle C^\top L, dL\rangle.Transpose inside the trace, then cycle; L⊤C=(C⊤L)⊤L^\top C = (C^\top L)^\top.
  4. dℓ=⟨(C+C⊤)L,dL⟩=⟨2CL,dL⟩d\ell = \langle(C + C^\top)L, dL\rangle = \langle 2CL, dL\rangle, so over the free entries ∇Lℓ=tril⁡(2CL)\nabla_L\ell = \operatorname{tril}(2CL).Trace trick; CC is symmetric; and as in Problem 8 only the lower triangle is parametrised.
  5. 2CL=−NΣ−1L+Σ−1SΣ−1L=−NL−⊤+L−⊤L−1SL−⊤2CL = -N\Sigma^{-1}L + \Sigma^{-1}S\Sigma^{-1}L = -NL^{-\top} + L^{-\top}L^{-1}SL^{-\top}.Σ−1=L−⊤L−1\Sigma^{-1} = L^{-\top}L^{-1} and L−1L=IL^{-1}L = I, applied to each term: Σ−1L=L−⊤\Sigma^{-1}L = L^{-\top}, and Σ−1SΣ−1L=L−⊤L−1S L−⊤\Sigma^{-1}S\Sigma^{-1}L = L^{-\top}L^{-1}S\,L^{-\top}.
  6. 2CL=L−⊤(M−NI)2CL = L^{-\top}(M - NI) with M=L−1SL−⊤M = L^{-1}SL^{-\top}.Factor L−⊤L^{-\top} out on the left of both terms.
  7. At Σ=S/N\Sigma = S/N: LL⊤=S/NLL^\top = S/N, so M=L−1(NLL⊤)L−⊤=NIM = L^{-1}(NLL^\top)L^{-\top} = NI and 2CL=02CL = 0.L−1L=IL^{-1}L = I and L⊤L−⊤=IL^\top L^{-\top} = I. Then the lower triangle of 00 is 00.
  8. ∇Lℓ=tril⁡(2 ∇Σℓ L)=tril⁡(L−⊤(M−NI))\nabla_L\ell = \operatorname{tril}\big(2\,\nabla_\Sigma\ell\,L\big) = \operatorname{tril}\big(L^{-\top}(M - NI)\big) with M=L−1SL−⊤M = L^{-1}SL^{-\top}; it is 00 at Σ=S/N\Sigma = S/NMM is the scatter matrix of the whitened data L−1(xn−μ)L^{-1}(x_n - \mu), and the gradient says: move LL until the whitened scatter is NN times the identity, that is, until Σ\Sigma explains the data's spread exactly. The maximum-likelihood covariance from the Gaussian page is unchanged by the reparametrisation, as it must be; what changes is the gradient the optimiser sees, which here needs two triangular solves and no inverse.

Problem 10

Let AA be symmetric with distinct eigenvalues λi\lambda_i and orthonormal eigenvectors qiq_i, and let EE be symmetric. Show that along A+tEA + tE each eigenvalue has derivative dλidt=qi⊤Eqi\tfrac{d\lambda_i}{dt} = q_i^\top Eq_i at t=0t = 0, and use it to recover ddtlog⁡det⁡(A+tE)=tr⁡(A−1E)\tfrac{d}{dt}\log\det(A + tE) = \operatorname{tr}(A^{-1}E) for positive definite AA.

Take as known that for distinct eigenvalues, λi(t)\lambda_i(t) and a unit eigenvector qi(t)q_i(t) of A+tEA + tE can be chosen to vary differentiably in tt.

  1. (A+tE) qi(t)=λi(t) qi(t)(A + tE)\,q_i(t) = \lambda_i(t)\,q_i(t).The eigenvector equation at every tt.
  2. Eqi+Aqi′=λi′qi+λiqi′Eq_i + Aq_i' = \lambda_i'q_i + \lambda_iq_i' at t=0t = 0.Differentiate both sides with respect to tt by the product rule; ddt(A+tE)=E\tfrac{d}{dt}(A + tE) = E. Primes are tt-derivatives at 00.
  3. qi⊤Eqi+qi⊤Aqi′=λi′+λi qi⊤qi′q_i^\top Eq_i + q_i^\top Aq_i' = \lambda_i' + \lambda_i\,q_i^\top q_i'.Multiply on the left by qi⊤q_i^\top; qi⊤qi=1q_i^\top q_i = 1.
  4. qi⊤Aqi′=(Aqi)⊤qi′=λi qi⊤qi′q_i^\top Aq_i' = (Aq_i)^\top q_i' = \lambda_i\,q_i^\top q_i'.AA is symmetric, so qi⊤A=(Aqi)⊤q_i^\top A = (Aq_i)^\top, and Aqi=λiqiAq_i = \lambda_iq_i.
  5. λi′=qi⊤Eqi\lambda_i' = q_i^\top Eq_i.The two qi′q_i' terms in step 3 are equal by step 4 and cancel. The eigenvector's derivative never needed to be found.
  6. ddtlog⁡det⁡(A+tE)=∑iλi′λi=∑iqi⊤Eqiλi\tfrac{d}{dt}\log\det(A + tE) = \sum_i\dfrac{\lambda_i'}{\lambda_i} = \sum_i\dfrac{q_i^\top Eq_i}{\lambda_i}.log⁡det⁡=∑ilog⁡λi\log\det = \sum_i\log\lambda_i for positive eigenvalues, and ddtlog⁡λi=λi′/λi\tfrac{d}{dt}\log\lambda_i = \lambda_i'/\lambda_i.
  7. ∑iqi⊤Eqiλi=∑itr⁡(qiqi⊤λiE)=tr⁡((∑iqiqi⊤λi)E)=tr⁡(A−1E)\sum_i\dfrac{q_i^\top Eq_i}{\lambda_i} = \sum_i\operatorname{tr}\Big(\dfrac{q_iq_i^\top}{\lambda_i}E\Big) = \operatorname{tr}\Big(\Big(\sum_i\dfrac{q_iq_i^\top}{\lambda_i}\Big)E\Big) = \operatorname{tr}(A^{-1}E).qi⊤Eqi=tr⁡(qiqi⊤E)q_i^\top Eq_i = \operatorname{tr}(q_iq_i^\top E) by the cyclic property; the trace is linear; and ∑iqiqi⊤/λi=QΛ−1Q⊤=A−1\sum_i q_iq_i^\top/\lambda_i = Q\Lambda^{-1}Q^\top = A^{-1} is the eigendecomposition of the inverse.
  8. dλidt=qi⊤Eqi\tfrac{d\lambda_i}{dt} = q_i^\top Eq_i, and ddtlog⁡det⁡(A+tE)=∑iqi⊤Eqi/λi=tr⁡(A−1E)\tfrac{d}{dt}\log\det(A + tE) = \sum_i q_i^\top Eq_i/\lambda_i = \operatorname{tr}(A^{-1}E)Jacobi's formula from Problem 3, reached through the spectrum. The per-eigenvalue form says what the trace hides: a perturbation moves log⁡det⁡\log\det most through the smallest eigenvalues, with weight 1/λi1/\lambda_i along qiq_i, which is why a nearly singular covariance makes its log⁡det⁡\log\det gradient large along the nearly flat direction.

Where this goes wrong

1. Differentiating the inverse like a reciprocal

The scalar rule d(1/a)=−da/a2d(1/a) = -da/a^2 is well worn, and A−1A^{-1} looks like 1/A1/A.

  1. A−1A=IA^{-1}A = IRight so far: Problem 1, step 1.
  2. “A−1A^{-1} is 1/A1/A, so its derivative is −1/A2-1/A^2 times dAdA.”The analogy that causes the mistake: in the scalar rule the two factors of 1/a1/a sit together because numbers commute; matrices do not.
  3. d(A−1)=−A−2 dAd(A^{-1}) = -A^{-2}\,dAProblem 1 gives −A−1(dA)A−1-A^{-1}(dA)A^{-1}, with dAdA between the two inverses. The two agree only when dAdA commutes with AA, for instance A=B+θIA = B + \theta I differentiated in θ\theta (Problem 4), which is exactly the case in which the habit is formed. In Problem 5 the wrong form gives ∇AL=−A−2⊤gb⊤\nabla_A L = -A^{-2\top}gb^\top in place of −A−⊤g x⊤-A^{-\top}g\,x^\top: the right shape, and a gradient check fails on every entry.

2. Trace trick without the transpose

Once dfdf is inside a trace, the matrix next to dXdX looks like the gradient.

  1. df=−tr⁡(X−1ba⊤X−1 dX)df = -\operatorname{tr}\big(X^{-1}ba^\top X^{-1}\,dX\big) for f=a⊤X−1bf = a^\top X^{-1}bRight so far: Problem 2, step 5.
  2. “The gradient is whatever multiplies dXdX inside the trace.”The shortcut that causes the mistake: the trace trick reads tr⁡(C⊤dX)\operatorname{tr}(C^\top dX), so what multiplies dXdX is C⊤C^\top, not CC.
  3. ∇X(a⊤X−1b)=−X−1ba⊤X−1\nabla_X(a^\top X^{-1}b) = -X^{-1}ba^\top X^{-1}The gradient is the transpose, −X−⊤ab⊤X−⊤-X^{-\top}ab^\top X^{-\top} (Problem 2). Both are n×nn\times n, so no shape check objects, and for symmetric XX with a=ba = b they coincide, which is the case the Gaussian page uses. Test the general case on one entry: with X=IX = I, ∂f/∂Xij=−aibj\partial f/\partial X_{ij} = -a_ib_j, the (i,j)(i,j) entry of −ab⊤-ab^\top, not of −ba⊤-ba^\top.

3. The gradient of det taken to be the gradient of log det

A−⊤A^{-\top} is the answer everyone remembers, and the log⁡\log is easy to drop or to add.

  1. d(det⁡A)=det⁡(A)tr⁡(A−1dA)d(\det A) = \det(A)\operatorname{tr}(A^{-1}dA)Right so far: Jacobi's formula, Problem 3.
  2. “So the gradient of the determinant is the inverse transpose.”The slip that causes the mistake: the factor det⁡A\det A in front of the trace is treated as part of the bookkeeping, when it is the difference between det⁡\det and log⁡det⁡\log\det.
  3. ∇Adet⁡A=A−⊤\nabla_A\det A = A^{-\top}That is ∇Alog⁡det⁡A\nabla_A\log\det A; the determinant's gradient is det⁡(A)A−⊤\det(A)A^{-\top} (Problem 3). With A=2IA = 2I in n=4n = 4 dimensions, det⁡A=16\det A = 16 and ∂det⁡A/∂A11=8\partial\det A/\partial A_{11} = 8 (the cofactor), while A−⊤A^{-\top} has entry 12\tfrac12 there. The error is a factor of det⁡A\det A on every entry, which in a likelihood is the difference between an update that scales with the volume of the covariance and one that does not.

4. Cholesky gradient left above the diagonal

Autodiff produces 2Σ−1L2\Sigma^{-1}L for the full matrix, and it is tempting to use all of it.

  1. dlog⁡det⁡Σ=⟨2Σ−1L,dL⟩d\log\det\Sigma = \langle 2\Sigma^{-1}L, dL\rangle over all n2n^2 entries of LLRight so far: Problem 8, step 7.
  2. “LL is a matrix parameter, so update every entry of LL with this gradient.”The shortcut that causes the mistake: 2Σ−1L=2L−⊤2\Sigma^{-1}L = 2L^{-\top} is upper triangular, and its nonzero off-diagonal entries sit exactly where LL has no parameters.
  3. L←L−η⋅2L−⊤L \leftarrow L - \eta\cdot 2L^{-\top}, so the entries above the diagonal of LL become nonzeroLL is no longer lower triangular, log⁡det⁡(LL⊤)=2∑ilog⁡Lii\log\det(LL^\top) = 2\sum_i\log L_{ii} no longer holds, and the next Cholesky-based solve is wrong. The gradient over the free entries is tril⁡(2L−⊤)\operatorname{tril}(2L^{-\top}), which is only the diagonal (Problem 8); a parametrisation that stores the lower triangle as a vector, or masks the gradient with tril⁡\operatorname{tril}, never sees the problem.

5. Chain rule through LLᵀ counting L once

Σ=LL⊤\Sigma = LL^\top is a product, but it is a product of a matrix with itself.

  1. ∇Σℓ=C\nabla_\Sigma\ell = C, symmetricRight so far: the Gaussian page.
  2. “Σ=LL⊤\Sigma = LL^\top is linear in LL with L⊤L^\top fixed, so by the chain rule ∇Lℓ=CL\nabla_L\ell = CL.”The shortcut that causes the mistake: dΣ=(dL)L⊤+L(dL)⊤d\Sigma = (dL)L^\top + L(dL)^\top has two terms, and the second is dropped because L⊤L^\top is treated as a different variable.
  3. ∇Lℓ=tril⁡(CL)\nabla_L\ell = \operatorname{tril}(CL)The two terms are equal (Problem 9, steps 2 and 3), so the gradient is tril⁡(2CL)\operatorname{tril}(2CL): the error is a factor of 22 on every entry. The stationary point is unchanged, which is why training can still reach the maximum-likelihood covariance, only with half the step the learning rate promises; the same slip in log⁡det⁡Σ\log\det\Sigma alone gives 1/Lii1/L_{ii} in place of 2/Lii2/L_{ii} (Problem 8).

Print this set: matrix-inverse-and-log-determinant-derivatives.pdf (problems, answers, and worked solutions on separate pages).

More practice sets

All practice sets

eulerpad.app/practice/matrix-inverse-and-log-determinant-derivatives