Guides / Matrix calculus

Backprop for a one-hidden-layer network: δ₂ = ŷ − y, δ₁ = σ′(z₁) ⊙ W₂ᵀδ₂, and every gradient shaped like its weight

The backward pass of a small classifier is four rules applied in order. Softmax with cross-entropy gives the output delta ŷ − y. Each weight matrix gets the delta at its output times its input, transposed. The delta moves down a layer through the transposed weights, then through the activation's derivative. And every gradient has the shape of the thing it updates, which catches most errors before any number is checked. The conventions are those of the matrix calculus conventions guide, with one column per example and z = Wx + b. The mistakes come from writing the outer product the wrong way round, from skipping the activation's derivative, and from leaving the bias gradient unsummed over a batch.

Why it works

Entry W_2,ij appears in only one output, z_2,i, multiplied by h_j, so its gradient is δ_2,ih_j: the row index belongs to the delta and the column index to the input, which is the outer product δ₂h^T. Hidden unit j feeds every output through column j of W₂, so its gradient collects Σ_i W_2,ijδ_2,i, which is W₂^Tδ₂.

That is the gradient at h, not at z₁. The sigmoid sits in between, and its Jacobian is diagonal, so one more elementwise product by σ′(z₁) gives δ₁. From there the first layer repeats the second's pattern with x as the input. In a batch, the parameters are shared by every example, so their gradients add up the examples' contributions, and the 1/N of the mean enters once, in Δ₂.

Example 1: the forward pass and the output delta

Take x = (1, −1), W₁ = 1121, W₂ = 1−102, both biases 0, and target class 1, so y = (1, 0). Find ŷ, L and δ₂.

1. z₁ = W₁x = (0, 1)1 − 1 and 2 − 1.
2. h = σ(z₁) ≈ (0.5, 0.7311)Save h: the backward pass needs it twice.
3. z₂ = W₂h ≈ (−0.2311, 1.4621)0.5 − 0.7311 and 2 × 0.7311.
4. ŷ ≈ (0.1554, 0.8446)Softmax of z₂. The network prefers the wrong class.
5. L = −log ŷ₁ ≈ 1.8620Cross-entropy keeps only the target's term.
6. δ₂ = ŷ − y ≈ (−0.8446, 0.8446)The gradient at z₂. It sums to 0, and its negative entry is the class to push up.

Example 2: the output layer's gradients

With the values of Example 1, find ∇_W2L, ∇_b2L and ∇_hL.

1. ∇_W2L = δ₂h^T(2 × 1)(1 × 2): the shape of W₂, which is k × m.
2. ≈ −0.4223−0.61750.42230.6175Row i is δ_2,i times h^T = (0.5, 0.7311).
3. ∇_b2L = δ₂Each bias enters one output with coefficient 1.
4. ∇_hL = W₂^Tδ₂The rows of W₂^T are (1, 0) and (−1, 2).
5. ≈ (−0.8446, 2.5339)Unit 2 feeds class 2 with weight 2, so it gets the larger share of the blame. Finite differences of L agree with all three gradients.

Example 3: the hidden delta and the first layer

Continue to δ₁, ∇_W1L and ∇_xL.

1. σ′(z₁) = h ⊙ (1 − h)From the saved h …
2. ≈ (0.25, 0.1966)… with no exponential recomputed.
3. δ₁ = σ′(z₁) ⊙ ∇_hLThrough the sigmoid, entry by entry.
4. ≈ (−0.2112, 0.4982)0.25 × (−0.8446) and 0.1966 × 2.5339.
5. ∇_W1L = δ₁x^TThe pattern of Example 2, one layer down.
6. ≈ −0.21120.21120.4982−0.4982Row i is δ_1,i times x^T = (1, −1); ∇_b1L = δ₁.
7. ∇_xL = W₁^Tδ₁ ≈ (0.7852, 0.2870)No update uses it, but it is what a layer below would receive. Every gradient here has the shape of its variable and matches a central finite difference.

Example 4: a batch of two

Add a second example, x′ = (0, 0) in class 2, and use the mean loss. With rows as examples, find Δ₂, ∇_W2L and ∇_b2L.

1. h′ = (0.5, 0.5)For x′ = 0, z₁ = 0.
2. z₂′ = (0, 1)W₂h′.
3. ŷ′ ≈ (0.2689, 0.7311)Softmax of (0, 1), so its delta is (0.2689, −0.2689).
4. Δ₂ = (Ŷ − Y)/2The two deltas as rows, each divided by N = 2 …
5. ≈ −0.42230.42230.1345−0.1345… once, here, and never again.
6. ∇_W2L = Δ₂^TH(2 × N)(N × 2): the mean of the two outer products, k × m again.
7. ≈ −0.1439−0.24150.14390.2415The second example pulls against the first.
8. ∇_b2L = Δ₂^T1 ≈ (−0.2879, 0.2879)The column sums of Δ₂: one bias is shared by the batch, so its gradient adds the rows. Finite differences of the mean loss agree.

Common mistakes, and the exact line they happen on

1. The outer product the wrong way round

The weight gradient of Example 2.

1. δ₂ ≈ (−0.8446, 0.8446)Right so far, with h ≈ (0.5, 0.7311).
2. ∇_W2L = hδ₂^TThis is the mistake. Writing the input first, in the order the data flows, gives the transpose: −0.42230.4223−0.61750.6175. Here m = k = 2, so the shape can't warn you. In δ_2,ih_j the row index is the output, so the delta comes first.

2. Skipping the activation's derivative

The hidden delta of Example 3.

1. ∇_hL ≈ (−0.8446, 2.5339)Right so far.
2. δ₁ = W₂^Tδ₂This is the mistake. That is the gradient at h. The sigmoid between h and z₁ multiplies each entry by σ′(z₁), at most 1/4, so this is 4 and 5.1 times too large. The delta is (−0.2112, 0.4982).

3. A bias gradient that is still a matrix

The batch of Example 4.

1. Δ₂, 2 × 2Right so far: one row per example.
2. ∇_b2L = Δ₂This is the mistake. For one example the bias gradient is the delta, but in a batch the same b₂ is added to every row, so its gradient sums them: (−0.2879, 0.2879). Array code may broadcast b₂ − ηΔ₂ without complaint and quietly give each example its own bias.

Practice

1. A network has 784 inputs, 128 hidden units and 10 classes. What are the shapes of W₁, W₂ and their gradients?

   Show answer

   W₁ and its gradient are 128 × 784; W₂ and its gradient are 10 × 128.

2. The output is ŷ = (0.1, 0.7, 0.2) and the target is class 2. Find δ₂.

   Show answer

   ŷ − y = (0.1, −0.3, 0.2).

3. Find ∇_W2L for δ₂ = (0.2, −0.2) and h = (1, 0.5).

   Show answer

   δ₂h^T: rows (0.2, 0.1) and (−0.2, −0.1).

4. With ReLU hidden units, z₁ = (1.2, −0.3, 0.5) and W₂^Tδ₂ = (0.4, 0.9, −0.6). Find δ₁.

   Show answer

   (0.4, 0, −0.6): the mask zeroes the unit with negative pre-activation.

5. The loss adds (λ/2)‖W₂‖_F² with λ = 0.01. What does it add to the gradient of an entry of W₂ equal to 3?

   Show answer

   λ × 3 = 0.03: the penalty's gradient is λW₂, since the 1/2 cancels the 2.