Guides / Matrix calculus

Matrix calculus conventions: numerator layout, gradients shaped like the variable, and the derivative read off the differential

Most wrong answers in matrix calculus are not calculus errors. They are layout errors: a transpose in the wrong place, or a product whose shapes don't multiply. The cure is to fix one convention and keep it. In numerator layout the derivative of a vector output has one row per output and one column per input, the gradient of a scalar is a column shaped like the variable, and a matrix gradient is shaped like the matrix. With those three rules, every derivative can be read off a differential, and the shapes check the answer for you.

Short answer

Vectors are columns. For y ∈ ℝ^m and x ∈ ℝⁿ, the Jacobian ∂y/∂x is m × n, with entry ∂y_i/∂x_j. For a scalar f, ∂f/∂x is a 1 × n row and the gradient ∇f is its transpose, an n × 1 column. Read it off the differential: if df = g^Tdx, then ∇f = g; if df = tr(G^TdX), then ∇_Xf = G, shaped like X. The standard results: ∂(Ax)/∂x = A, ∇(a^Tx) = a, ∇(x^TAx) = (A + A^T)x and ∇_W(a^TWb) = ab^T.

Why it works

A derivative is the best linear approximation: a small change dx in the input makes the change dy ≈ J dx in the output. For that product to make sense, J needs one column for each entry of x and one row for each entry of y, which is numerator layout. A scalar f has one output, so its derivative is a single row, and df = (∂f/∂x) dx is a row times a column.

The gradient is that row stood up as a column, so that it has the shape of x and a gradient step x − η∇f makes sense. The same reasoning gives a matrix variable a gradient with its own shape, and the trace turns the sum Σ G_ij dX_ij into matrix form: tr(G^TdX) = Σ_ij G_ijdX_ij. Once a differential is in one of these forms, the gradient is whatever sits in the slot.

Example 1: a Jacobian from its entries

Let y = (x₁² + x₂, x₁x₂, 3x₂). Find ∂y/∂x at x = (1, 2), and its shape.

y ∈ ℝ³, x ∈ ℝ²Three outputs, two inputs, so the Jacobian is 3 × 2.
row 1: (2x₁, 1)The partials of y₁ = x₁² + x₂ in x₁, then x₂.
row 2: (x₂, x₁)The partials of y₂ = x₁x₂.
row 3: (0, 3)y₃ = 3x₂ doesn't depend on x₁.
J = 212103At (1, 2). Rows follow the outputs, columns the inputs. A small step dx changes y by about J dx: dx = (0.01, 0) moves y by (0.02, 0.02, 0).

Example 2: a gradient from the differential

Let f(x) = (a^Tx)² with a = (1, 2). Find ∇f, then evaluate it at x = (3, 1).

df = 2(a^Tx) d(a^Tx)The scalar chain rule: a^Tx is a number, and f is its square.
d(a^Tx) = a^Tdxa is constant.
df = (2(a^Tx)a)^T dxThe scalar 2a^Tx can sit on either side, so the differential is in the form g^Tdx.
∇f = 2(a^Tx) aRead g off the slot. It is a column, the shape of x.
a^Tx = 3 + 2 = 5At x = (3, 1).
∇f = 10a = (10, 20)Check by differences: f(3.001, 1) − f(2.999, 1) is about 0.02, and 0.02/0.002 = 10.

Example 3: a gradient with respect to a matrix

Let f(W) = a^TWb with W a 2 × 3 matrix, a = (1, 2) and b = (3, −1, 2). Find ∇_Wf.

df = a^T(dW)bf is linear in W, so its differential has the same form.
= tr(a^TdW b)A number is its own trace.
= tr(ba^TdW)The trace doesn't change when a product is cycled: move b to the front.
= tr((ab^T)^TdW)ba^T = (ab^T)^T, so the differential is in the form tr(G^TdW).
∇_Wf = ab^TRead G off the slot: a 2 × 1 times a 1 × 3, which is 2 × 3, the shape of W.
= 3−126−24Entry (i, j) is a_ib_j, the coefficient of W_ij in Σ a_iW_ijb_j.

Example 4: the same derivative in the other layout

Let A = 123456 and c = (1, 0, −1). Find the Jacobian of y = Ax and the gradient of f = c^TAx, in numerator layout and in the denominator layout some textbooks use.

∂y/∂x = A, 3 × 2Numerator layout: one row per entry of y.
A^T, 2 × 3Denominator layout lays out the same partials transposed, one row per entry of x.
∂f/∂x = c^TA, 1 × 2Numerator layout: the row derivative.
∇f = A^TcThe gradient is the transposed row, a column.
= (1 − 5, 2 − 6)Rows of A^T times c.
∇f = (−4, −4)In denominator layout ∂f/∂x is this column already. The two layouts disagree about Jacobians and agree about gradients, which always take the shape of the variable.

Common mistakes, and the exact line they happen on

1. Writing 2Ax for a matrix that isn't symmetric

Find ∇(x^TAx) at x = (1, 1) for A = 1201.

f = x^TAxRight so far: f = x₁² + 2x₁x₂ + x₂².
∇f = 2Ax = (6, 2)This is the mistake. x appears twice, and the two copies give Ax + A^Tx. They match only if A = A^T. Here (A + A^T)x = (4, 4), and the partials of the expanded f agree: 2x₁ + 2x₂ = 4 both times.

2. A matrix gradient in the wrong shape

The gradient of Example 3, f(W) = a^TWb.

df = a^T(dW)bRight so far.
∇_Wf = ba^TThis is the mistake. ba^T is what multiplies dW inside the trace, which is G^T, not G. It is 3 × 2, and W is 2 × 3, so W − η ba^T isn't defined. The gradient must have W's shape: ab^T.

3. Taking a Jacobian from a denominator-layout table

The map y = Ax of Example 4, with A a 3 × 2 matrix.

∇(c^TAx) = A^TcRight so far: a gradient is the same in both layouts.
∂(Ax)/∂x = A^TThis is the mistake. That is the denominator-layout Jacobian copied into a numerator-layout derivation. It is 2 × 3, so dy ≈ A^Tdx multiplies a 2 × 3 by a 2 × 1, which isn't defined. In numerator layout the Jacobian is A. Pick one layout and convert every formula you borrow.

Practice

For y ∈ ℝ⁴ and x ∈ ℝ⁷, what shape is ∂y/∂x in numerator layout?
Show answer
4 × 7: one row per output.
Find ∇(x^Tx) at x = (1, −2, 2).
Show answer
2x = (2, −4, 4).
Find ∇(a^Tx + ‖x‖²).
Show answer
a + 2x.
Find ∇(x^TAx) at x = (1, 1) for A = 0130.
Show answer
(A + A^T)x = (4, 4). Not 2Ax = (2, 6).
W and C have the same shape. Find ∇_W tr(W^TC).
Show answer
C: tr(W^TC) = Σ W_ijC_ij.