Guides / Matrix calculus

Batched backprop for a dense layer: dW = XᵀdY, db = dYᵀ1, dX = dY Wᵀ, and the 1/N applied once

In array code a dense layer takes a whole minibatch at once: Y = XW + 1b^T, with one row per example and W stored inputs by outputs. Its backward pass is three matrix products, and each one can be read off from where the batch index sits. The weight gradient sums over the batch, so N is the inner dimension of X^TdY. The bias gradient sums the rows of dY. The input gradient doesn't sum over the batch at all. The conventions are those of the matrix calculus conventions guide: dA is the gradient of the loss with respect to A, with the shape of A. The one-hidden-layer backprop guide did the same network one example at a time with W stored outputs by inputs, so its formulas look transposed next to these. The mistakes come from carrying that layout over, from dividing by N a second time, and from multiplying by W where W^T belongs.

Why it works

In index form, Y_nj = Σ_i X_niW_ij + b_j. The weight W_ij appears in column j of every row, multiplied by X_ni, so its gradient adds up X_ni dY_nj over the examples n. A sum over a shared index is a matrix product, and the shared index is the batch, so it has to be the inner dimension: X^TdY. The same sum, with coefficient 1, gives the bias its column sums.

The input is different. X_ni appears only in row n of Y, once per output feature, so its gradient sums over the features, Σ_j dY_njW_ij, and every example keeps its own row. That is the only place W is used on the way back, and it runs from m outputs back to d inputs, which is what W^T does. Checking that each result has the shape of its variable catches most slips, as long as d ≠ m.

Example 1: the forward pass and its shapes

Take three examples with two features, X = 1201−11, two outputs with W = 1−120, and b = (0, 1). Find Y.

1. XW is (3 × 2)(2 × 2) = 3 × 2N = 3, d = 2, m = 2. W has one row per input feature.
2. row 1: (1, 2)W = (5, −1)1 × (1, −1) + 2 × (2, 0): a row of X combines the rows of W.
3. rows 2 and 3: (2, 0) and (1, 1)(0, 1)W is W's second row; (−1, 1)W = (−1, 1) + (2, 0).
4. 1b^T adds (0, 1) to every rowThe bias is broadcast: each example uses the same b.
5. Y = 502112Row n is x^(n)TW + b^T, the layer applied to example n alone. The batch is three copies of one layer sharing W and b.

Example 2: dW, db and dX from an upstream gradient

The layer above sends back dY = 100−121. Find dW, db and dX.

1. dW = X^TdY(2 × 3)(3 × 2) = 2 × 2, the shape of W. The batch index is summed away.
2. row 1: (1, 0) − (2, 1)Row i of X^TdY weights the rows of dY by column i of X, here (1, 0, −1) …
3. dW = −1−140… and row 2 is 2(1, 0) + 1(0, −1) + 1(2, 1) = (4, 0).
4. db = dY^T1 = (3, 0)The column sums of dY: 1 + 0 + 2 and 0 − 1 + 1.
5. dX = dY W^T(3 × 2)(2 × 2) = 3 × 2, the shape of X. W^T has rows (1, 2) and (−1, 0).
6. dX = 121014Row 1 is 1 × (1, 2) + 0 × (−1, 0). Each row uses only its own row of dY. Central finite differences agree with all three gradients.

Example 3: a mean squared loss, with the 1/N in one place

Keep X, W and b, and use L = ‖Y − T‖_F²/(2N) with targets T = 402211. Find dY, dW and db, and check dW against the per-example outer products.

1. Y − T = 100−101So L = (1 + 1 + 1)/6 = 0.5.
2. dY = (Y − T)/N = (Y − T)/3The 1/N of the mean enters here, once.
3. dW = X^TdYThe rule of Example 2, which never asked where dY came from …
4. dW = 1/3−1/32/30… so it carries the 1/3 without applying it again.
5. Σ_n x⁽ⁿ⁾dy^(n)T gives the same matrixExample 1 contributes (1/3)(1, 2)^T(1, 0), example 2 contributes (1/3)(0, 1)^T(0, −1), example 3 adds the rest.
6. db = dY^T1 = (1/3, 0)A plain sum: the average is already built in. Finite differences of L agree with dW and db.

Example 4: micro-batches of unequal size

Memory allows only two examples at a time, so the batch of Example 3 is split into examples 1–2 and example 3, each with its own mean loss. Combine the two weight gradients into the full-batch one.

1. g₁ = 0.501−0.5Example 3's rule on the first two rows, with 1/2 in place of 1/3.
2. g₂ = 0−101One row, so the mean is the row itself: (−1, 1)^T(0, 1).
3. L = (2/3)L₁ + (1/3)L₂Each mean is weighted by its share of the N = 3 examples.
4. dW = (2/3)g₁ + (1/3)g₂The gradient of a weighted sum.
5. dW = 1/3−1/32/30Example 3's answer. The plain average (g₁ + g₂)/2 gives 0.25−0.50.50.25, which overweights the lone example. It is right only when the sizes are equal.

Common mistakes, and the exact line they happen on

1. The weight gradient in the other layout

The weight gradient of Example 2.

1. Y = XW + 1b^T, W is d × mRight so far.
2. dW = dY^TXThis is the mistake. That is the rule for W stored outputs by inputs, as in XW^T. Here it gives −14−10, the transpose of the answer. With d = m the update runs anyway. W_ij multiplies X_ni, so X goes on the left: X^TdY.

2. Dividing by N twice

The bias gradient of Example 3.

1. dY = (Y − T)/3Right so far: the mean's 1/N is inside dY.
2. db = dY^T1/N = (1/9, 0)This is the mistake. "The bias gradient is the average over the batch" describes the result, not the formula. Averaging a dY that already carries 1/N makes the bias learn N times too slowly. The answer is the plain sum, (1/3, 0).

3. W where Wᵀ belongs

The input gradient of Example 2.

1. dX must be 3 × 2, like XRight so far.
2. dX = dY WThis is the mistake. The forward pass multiplies by W, but the backward pass runs from outputs to inputs. With a square W the product still runs, giving rows (1, −1), (−2, 0) and (4, −2) instead of (1, 2), (1, 0) and (1, 4). The sum is over W's second index, so it is dY W^T.

Practice

1. A layer maps 784 features to 128 on a batch of 64. What are the shapes of dW, db and dX?

   Show answer

   dW is 784 × 128, db has length 128 and dX is 64 × 784.

2. Find dW for X = 1021 and dY = 1−102.

   Show answer

   X^TdY has rows (1, 3) and (0, 2).

3. With a mean loss over two examples, dY has rows (0.5, 0) and (−0.25, 0.25). Find db.

   Show answer

   (0.25, 0.25), the column sums. The 1/2 is already inside dY.

4. W is 2 × 3 with rows (1, 2, 0) and (0, 1, −1). One row of dY is (1, 0, 2). Find that row of dX.

   Show answer

   (1, −2): the dot product of (1, 0, 2) with each row of W.

5. A batch of 8 is run as micro-batches of 6 and 2, each with its own mean loss and gradients g₁ and g₂. What is the full-batch gradient?

   Show answer

   (3/4)g₁ + (1/4)g₂, the weights 6/8 and 2/8.