The multivariate Gaussian: gradients and identities

Worked solutions

Problem 1

1. $\log\mathcal N(x; \mu, \Sigma) = -\tfrac d2\log(2\pi) - \tfrac12\log\det\Sigma - \tfrac12(x - \mu)^\top\Sigma^{-1}(x - \mu)$The log of a product is the sum of the logs, $\log c^{-1/2} = -\tfrac12\log c$, and $\log$ undoes $\exp$. The first two terms do not depend on $x$: they are what make the density integrate to $1$.
2. $-\tfrac d2\log(2\pi) = -\log(2\pi)$$d = 2$.
3. $\det\Sigma = 2 \cdot 2 - 1 \cdot 1 = 3$The determinant of a $2 \times 2$ matrix is $ad - bc$.
4. $\Sigma^{-1} = \tfrac13\begin{pmatrix} 2 & -1 \\ -1 & 2 \end{pmatrix}$For a $2 \times 2$ matrix, swap the diagonal entries, negate the off-diagonal ones and divide by the determinant; multiplying by $\Sigma$ gives $I$.
5. $x - \mu = (1, 1)^\top$, $\Sigma^{-1}(x - \mu) = \tfrac13(1, 1)^\top$ and $q(x) = \tfrac13(1 + 1) = \tfrac23$Each row of the matrix in step 4 sums to $2 - 1 = 1$, and $q$ is the dot product of $x - \mu$ with that vector.
6. $\log\mathcal N(x; \mu, \Sigma) = -\log(2\pi) - \tfrac12\log 3 - \tfrac13 \approx -2.721$Steps 2 to 5 in step 1: $-1.838 - 0.549 - 0.333$. Only the last term depends on $x$, and it depends on $x$ only through the Mahalanobis distance.

Problem 2

1. Only $-\tfrac12 q(x)$ depends on $x$.The first two terms of Problem 1, step 1 are constants, and a constant has zero gradient.
2. With $v = x - \mu$, $\nabla_v(v^\top\Sigma^{-1}v) = 2\Sigma^{-1}v$.$\nabla_v(v^\top Av) = (A + A^\top)v$ (the matrix-calculus page, Problem 4), and $\Sigma^{-1}$ is symmetric because $(\Sigma^{-1})^\top = (\Sigma^\top)^{-1} = \Sigma^{-1}$.
3. $\nabla_x q = 2\Sigma^{-1}(x - \mu)$$\partial v/\partial x = I$: subtracting a constant does not change a derivative.
4. With $\Sigma = \operatorname{diag}(\sigma_i^2)$, $\Sigma^{-1} = \operatorname{diag}(1/\sigma_i^2)$ and entry $i$ of the score is $-(x_i - \mu_i)/\sigma_i^2$.The inverse of a diagonal matrix inverts each diagonal entry. Each coordinate gets the one-dimensional score, because with a diagonal $\Sigma$ the density is a product of $d$ one-dimensional densities and its log a sum.
5. $\nabla_x\log\mathcal N(x; \mu, \Sigma) = -\Sigma^{-1}(x - \mu)$, which is $(-\tfrac13, -\tfrac13)^\top$ at Problem 1's numbersStep 3 times $-\tfrac12$; Problem 1, step 5 gives $\Sigma^{-1}(x - \mu)$. The score points back towards $\mu$, weighting the directions of small variance most. Here $x - \mu = (1, 1)^\top$ is an eigenvector of $\Sigma$ with eigenvalue $3$, so the score points straight back at $\mu$, shrunk by a factor $3$. Score-based diffusion models train a network to approximate this vector field.

Problem 3

1. $\nabla_\mu\log\mathcal N(x^{(n)}; \mu, \Sigma) = \Sigma^{-1}(x^{(n)} - \mu)$Problem 2's computation with $v = x^{(n)} - \mu$, whose Jacobian with respect to $\mu$ is $-I$; that flips the sign.
2. $\nabla_\mu\ell = \Sigma^{-1}\sum_n(x^{(n)} - \mu) = N\Sigma^{-1}(\bar x - \mu)$The gradient of a sum is the sum of the gradients, $\Sigma^{-1}$ factors out of the sum, and $\sum_n x^{(n)} = N\bar x$.
3. $\nabla_\mu\ell = 0 \iff \mu = \bar x$$\Sigma^{-1}$ is invertible, so it sends only the zero vector to zero.
4. $\nabla_\mu^2\ell = -N\Sigma^{-1}$, which is negative definite.Step 2 is affine in $\mu$, so its Jacobian is the matrix that multiplies $\mu$. $\Sigma^{-1}$ is positive definite, so $\ell$ is strictly concave in $\mu$ and its stationary point is the global maximum.
5. $\nabla_\mu\ell = N\Sigma^{-1}(\bar x - \mu)$, so $\hat\mu = \bar x$ for every $\Sigma$$\Sigma$ dropped out in step 3, so the mean can be estimated before the covariance is known. That is what lets Problem 6 put $\bar x$ in place of $\mu$.

Problem 4

1. $\det X = \sum_j X_{ij}C_{ij}$ for any fixed row $i$, where $C_{ij}$ is $(-1)^{i+j}$ times the determinant of $X$ with row $i$ and column $j$ deleted.The cofactor (Laplace) expansion along row $i$.
2. $\partial\det X/\partial X_{ij} = C_{ij}$Every cofactor in the row-$i$ expansion deletes row $i$, so $X_{ij}$ appears only as the explicit factor of the $j$-th term.
3. $X^{-1} = C^\top/\det X$The inverse is the adjugate divided by the determinant, and the adjugate is the transpose of the matrix of cofactors.
4. $\partial\log\det X/\partial X_{ij} = C_{ij}/\det X = (X^{-1})_{ji}$Chain rule through the log, whose derivative is $1/\det X$; then step 3 read at entry $(j, i)$.
5. $\nabla_X\log\det X = X^{-\top}$, so $\nabla_\Sigma\log\det\Sigma = \Sigma^{-1}$Step 4 puts entry $(j, i)$ of $X^{-1}$ at position $(i, j)$, which is the transpose. For a symmetric $\Sigma$, $\Sigma^{-\top} = \Sigma^{-1}$. In differential form, $d\log\det\Sigma = \operatorname{tr}(\Sigma^{-1}\,d\Sigma)$, which Problem 6 uses.

Problem 5

1. $d(\Sigma^{-1}) = -\Sigma^{-1}\,d\Sigma\,\Sigma^{-1}$Differentiate $\Sigma\Sigma^{-1} = I$ with the product rule, $d\Sigma\,\Sigma^{-1} + \Sigma\,d(\Sigma^{-1}) = 0$, and multiply on the left by $\Sigma^{-1}$. The factors keep their order because matrices do not commute.
2. $d\operatorname{tr}(\Sigma^{-1}A) = -\operatorname{tr}(\Sigma^{-1}\,d\Sigma\,\Sigma^{-1}A) = \operatorname{tr}\big(-\Sigma^{-1}A\Sigma^{-1}\,d\Sigma\big)$The trace is linear, so the differential passes inside; cycling moves $\Sigma^{-1}A$ from the end to the front.
3. $\nabla_\Sigma\operatorname{tr}(\Sigma^{-1}A) = \big(-\Sigma^{-1}A\Sigma^{-1}\big)^\top = -\Sigma^{-1}A\Sigma^{-1}$$df = \operatorname{tr}(M\,d\Sigma)$ means the gradient is $M^\top$ (Before you start). Transposing reverses the order of a product, and $\Sigma^{-1}$ and $A$ are symmetric.
4. $v^\top\Sigma^{-1}v = \operatorname{tr}(\Sigma^{-1}vv^\top)$A scalar is its own trace, and the trace is cyclic; $vv^\top$ is symmetric.
5. $\nabla_\Sigma\operatorname{tr}(\Sigma^{-1}A) = -\Sigma^{-1}A\Sigma^{-1}$ and $\nabla_\Sigma(v^\top\Sigma^{-1}v) = -\Sigma^{-1}vv^\top\Sigma^{-1}$Step 3, then step 4 with $A = vv^\top$. With $d = 1$ it is the scalar rule $\tfrac{d}{ds}(a/s) = -a/s^2$. The minus sign says that a larger covariance makes every point closer in Mahalanobis distance.

Problem 6

1. $\ell(\Sigma) = -\tfrac{Nd}{2}\log(2\pi) - \tfrac N2\log\det\Sigma - \tfrac12\operatorname{tr}(\Sigma^{-1}S)$Sum Problem 1, step 1 over the $N$ examples. Each quadratic term is $\operatorname{tr}(\Sigma^{-1}vv^\top)$ (Problem 5, step 4), and a sum of traces is the trace of the sum.
2. $\nabla_\Sigma\ell = -\tfrac N2\Sigma^{-1} + \tfrac12\Sigma^{-1}S\Sigma^{-1}$Problem 4 for the log-determinant and Problem 5 with $A = S$, which is symmetric as a sum of outer products $vv^\top$.
3. $\nabla_\Sigma\ell = 0 \iff -N\Sigma + S = 0$Multiply on the left and on the right by $\Sigma$, then by $2$. $\Sigma$ is invertible, so this can be undone and the two equations have the same solutions.
4. With $\Lambda = \Sigma^{-1}$, $\ell = \tfrac N2\log\det\Lambda - \tfrac12\operatorname{tr}(\Lambda S) + \text{const}$, which is concave in $\Lambda$.$\log\det\Lambda = -\log\det\Sigma$. $\log\det$ is concave on positive definite matrices (a standard fact) and $\operatorname{tr}(\Lambda S)$ is linear, so the one stationary point is the global maximum. $S/N$ is positive definite because $S$ is a positive semidefinite sum of outer products and invertible.
5. $\nabla_\Sigma\ell = -\tfrac N2\Sigma^{-1} + \tfrac12\Sigma^{-1}S\Sigma^{-1}$, so $\hat\Sigma = \tfrac1N\sum_n(x^{(n)} - \bar x)(x^{(n)} - \bar x)^\top$It divides by $N$, not $N - 1$: it is NumPy's np.cov(X.T, bias=True), while the default np.cov divides by $N - 1$. The centred examples span at most $N - 1$ dimensions, so $S$ is invertible only if $N > d$; otherwise $\ell$ has no maximum, because $\Sigma$ can collapse onto the data's subspace and $\log\det\Sigma \to -\infty$.

Problem 7

1. $\mathbb E[y] = A\,\mathbb E[x] + b = A\mu + b$Expectation is linear, and $A$ and $b$ are constants.
2. $y - \mathbb E[y] = A(x - \mu)$$b$ cancels.
3. $\operatorname{Cov}[y] = \mathbb E\big[A(x - \mu)(x - \mu)^\top A^\top\big] = A\Sigma A^\top$$(Av)^\top = v^\top A^\top$, and the constants $A$ and $A^\top$ come out of the expectation on either side.
4. For $x_1 + x_2$: $A = (1, 1)$ and $b = 0$, so $A\mu = 1 - 1 = 0$ and $A\Sigma A^\top = 2 + 1 + 1 + 2 = 6$.$(1, 1)\,\Sigma\,(1, 1)^\top$ adds up every entry of $\Sigma$.
5. $y \sim \mathcal N(A\mu + b, A\Sigma A^\top)$; for Problem 1, $x_1 + x_2 \sim \mathcal N(0, 6)$A Gaussian is fixed by its mean and covariance, so steps 1 and 3 give the whole distribution. The variance $6$ exceeds $2 + 2$ by twice the covariance $\Sigma_{12} = 1$. With $A = (I\ \ 0)$ the same result says that a block of a Gaussian vector is Gaussian, with the matching blocks of $\mu$ and $\Sigma$; Problem 9 uses that.

Problem 8

1. $x \sim \mathcal N(\mu, LIL^\top) = \mathcal N(\mu, \Sigma)$Problem 7 with $A = L$, $b = \mu$, and $\varepsilon$ of mean $0$ and covariance $I$.
2. $f(x) = \mu^\top B\mu + 2\mu^\top BL\varepsilon + \varepsilon^\top L^\top BL\varepsilon$Expand $(\mu + L\varepsilon)^\top B(\mu + L\varepsilon)$. The two cross terms are equal scalars because $B$ is symmetric: $(L\varepsilon)^\top B\mu = \mu^\top BL\varepsilon$.
3. $F = \mu^\top B\mu + \operatorname{tr}(L^\top BL)$$\mathbb E[\varepsilon] = 0$ removes the middle term, and $\mathbb E[\varepsilon^\top M\varepsilon] = \mathbb E[\operatorname{tr}(M\varepsilon\varepsilon^\top)] = \operatorname{tr}(M\,\mathbb E[\varepsilon\varepsilon^\top]) = \operatorname{tr}(M)$.
4. $\nabla_\mu F = 2B\mu$ and $\nabla_L F = 2BL$$\nabla_x(x^\top Bx) = (B + B^\top)x$ and $\nabla_X\operatorname{tr}(X^\top BX) = (B + B^\top)X$ (the matrix-calculus page, Problems 4 and 8), with $B = B^\top$.
5. $\nabla f(x) = 2Bx$, so $\mathbb E[\nabla f(x)] = 2B\mu$ and $\mathbb E[\nabla f(x)\,\varepsilon^\top] = 2B\big(\mu\,\mathbb E[\varepsilon]^\top + L\,\mathbb E[\varepsilon\varepsilon^\top]\big) = 2BL$Substitute $x = \mu + L\varepsilon$ and use linearity, $\mathbb E[\varepsilon] = 0$ and $\mathbb E[\varepsilon\varepsilon^\top] = I$.
6. $\nabla_\mu F = 2B\mu = \mathbb E[\nabla f(x)]$ and $\nabla_L F = 2BL = \mathbb E[\nabla f(x)\,\varepsilon^\top]$This holds for any smooth $f$: $F = \mathbb E_\varepsilon[f(\mu + L\varepsilon)]$, the distribution of $\varepsilon$ does not involve the parameters, so the gradient passes inside the expectation, and $\partial f(\mu + L\varepsilon)/\partial L_{ij} = (\nabla f)_i\,\varepsilon_j$. That is why a VAE samples $\varepsilon$ rather than $x$: the average of $\nabla f(x)\,\varepsilon^\top$ over samples is an unbiased estimate of the gradient. Only the lower triangle of $L$ is free, so its gradient is the lower triangle of $2BL$.

Problem 9

1. $(z, x_b)$ is jointly Gaussian.It is the affine image of $x$ under $\begin{pmatrix} I & -K \\ 0 & I \end{pmatrix}$, so Problem 7 applies.
2. $\operatorname{Cov}[z, x_b] = \Sigma_{ab} - K\Sigma_{bb} = \Sigma_{ab} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{bb} = 0$The cross-covariance $\mathbb E[(z - \mathbb E z)(x_b - \mu_b)^\top]$ is linear in $z$, and $\operatorname{Cov}[x_a, x_b] = \Sigma_{ab}$, $\operatorname{Cov}[x_b, x_b] = \Sigma_{bb}$.
3. $z$ is independent of $x_b$.For jointly Gaussian vectors, zero cross-covariance makes the joint covariance block-diagonal, so the quadratic in the exponent and the determinant both split, and the density factorises.
4. $\mathbb E[z] = \mu_a - K\mu_b$ and $\operatorname{Cov}[z] = \Sigma_{aa} - K\Sigma_{ba} - \Sigma_{ab}K^\top + K\Sigma_{bb}K^\top = \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}$Problem 7 with $A = (I\ \ {-K})$. Each of the last three terms equals $\Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}$, because $\Sigma_{bb}^{-1}$ is symmetric and $\Sigma_{ab}^\top = \Sigma_{ba}$; two carry a minus sign and one a plus.
5. Given $x_b$, $x_a = z + Kx_b$ with $z$ still distributed as in step 4.By step 3, conditioning on $x_b$ does not change the distribution of $z$, and $Kx_b$ becomes a constant shift: the mean moves by $Kx_b$ and the covariance stays.
6. For Problem 1, $K = \Sigma_{12}/\Sigma_{22} = \tfrac12$, the mean is $1 + \tfrac12\big(0 - (-1)\big) = \tfrac32$ and the variance is $2 - \tfrac12 \cdot 1 = \tfrac32$.Every block is $1 \times 1$: $\mu_a = 1$, $\mu_b = -1$, $\Sigma_{aa} = \Sigma_{bb} = 2$ and $\Sigma_{ab} = 1$.
7. $x_a \mid x_b \sim \mathcal N\big(\mu_a + \Sigma_{ab}\Sigma_{bb}^{-1}(x_b - \mu_b),\ \Sigma_{aa} - \Sigma_{ab}\Sigma_{bb}^{-1}\Sigma_{ba}\big)$; for Problem 1, $x_1 \mid x_2 = 0 \sim \mathcal N(\tfrac32, \tfrac32)$The covariance is the Schur complement of $\Sigma_{bb}$ in $\Sigma$, which by the block-inverse formula is $(\Lambda_{aa})^{-1}$, the inverse of the precision's top-left block. It does not depend on the observed $x_b$ and is never larger than $\Sigma_{aa}$: observing a correlated variable can only narrow the distribution. Gaussian-process prediction is this formula.

Problem 10

1. $\log\big[\mathcal N(x; a, A)\,\mathcal N(x; b, B)\big] = -\tfrac12(x - a)^\top A^{-1}(x - a) - \tfrac12(x - b)^\top B^{-1}(x - b) + \text{const}$The log of a product is the sum of the logs, and the normalising constants do not depend on $x$.
2. $= -\tfrac12 x^\top(A^{-1} + B^{-1})x + x^\top(A^{-1}a + B^{-1}b) + \text{const}$$(x - a)^\top A^{-1}(x - a) = x^\top A^{-1}x - 2x^\top A^{-1}a + a^\top A^{-1}a$, the two cross terms being equal because $A^{-1}$ is symmetric; the same for $b$.
3. $-\tfrac12(x - c)^\top C^{-1}(x - c) = -\tfrac12 x^\top C^{-1}x + x^\top C^{-1}c + \text{const}$The same expansion.
4. $C^{-1} = A^{-1} + B^{-1}$ and $C^{-1}c = A^{-1}a + B^{-1}b$Two quadratics in $x$ with the same quadratic and linear parts differ by a constant, so the densities differ by a constant factor. $A^{-1} + B^{-1}$ is positive definite as a sum of positive definite matrices, so $C$ exists.
5. One dimension: $C = (\tfrac14 + \tfrac12)^{-1} = \tfrac43$ and $c = \tfrac43\big(\tfrac04 + \tfrac32\big) = 2$Step 4 with $1 \times 1$ matrices.
6. $\mathcal N(x; a, A)\,\mathcal N(x; b, B) \propto \mathcal N(x; c, C)$ with $C = (A^{-1} + B^{-1})^{-1}$, $c = C(A^{-1}a + B^{-1}b)$; for the 1-D numbers, $c = 2$ and $C = \tfrac43$Precisions add, and the new mean is a precision-weighted average: $2$ lies nearer $3$ than $0$ because $B = 2$ is the more certain of the two. The constant of proportionality is $\mathcal N(a; b, A + B)$. This is Bayes' rule for a Gaussian prior on a mean and one Gaussian measurement of it, the update step of a Kalman filter.