One-hidden-layer backprop, the whole backward pass

Worked solutions

Problem 1

1. $W_1 x \in \mathbb{R}^m$ for $x \in \mathbb{R}^d$, so $W_1$ is $m \times d$; $b_1$ is added to $W_1 x$, so $b_1$ and $z_1$ have $m$ entries.A matrix takes $d$-vectors to $m$-vectors only if it has $d$ columns and $m$ rows, and vectors can be added only if their shapes agree.
2. $h = \sigma(z_1)$ has $m$ entries.$\sigma$ is elementwise, so it keeps the shape of its input.
3. $W_2 h \in \mathbb{R}^k$, so $W_2$ is $k \times m$; $b_2$, $z_2$ and $\hat y$ have $k$ entries.Step 1 one layer up, with $h$ as the input; the softmax gives one probability per class, so it keeps the shape of $z_2$.
4. Entry $(i,j)$ of $\nabla_{W_1} L$ is $\partial L/\partial W_{1,ij}$, and likewise for every other variable.A gradient has one entry per entry of its variable, which is why $W_1 - \eta\,\nabla_{W_1} L$ is defined.
5. $W_1: m\times d$, $b_1, z_1, h: m$; $W_2: k\times m$, $b_2, z_2, \hat y: k$; each gradient has the shape of its variable: $\nabla_{W_1}L: m\times d$, $\nabla_{b_1}L: m$, $\nabla_{W_2}L: k\times m$, $\nabla_{b_2}L: k$, $\nabla_x L: d$Every answer below is checked against this list.

Problem 2

1. As a function of $z_2$, $L = -\sum_i y_i \log \operatorname{softmax}(z_2)_i$.$\hat y = \operatorname{softmax}(z_2)$, and $\delta_2$ holds everything below $z_2$ fixed.
2. Let $c$ be the index with $y_c = 1$. Then $L = -z_{2,c} + \log\sum_j e^{z_{2,j}}$.Only the target term of the sum survives, and $\log\hat y_c = z_{2,c} - \log\sum_j e^{z_{2,j}}$.
3. $\partial L/\partial z_{2,j} = -y_j + \hat y_j$.$z_{2,c}$ contains $z_{2,j}$ only when $j = c$, which is $y_j$; the log-sum-exp term differentiates to $e^{z_{2,j}}/\sum_{j'} e^{z_{2,j'}} = \hat y_j$. This is the softmax page's Problem 5 with $z = z_2$.
4. $\delta_2 = \hat y - y$$k \times 1$, the shape of $z_2$. Its entries sum to $0$: the target entry is $\hat y_c - 1 < 0$ and the others are positive.

Problem 3

1. $z_{2,i} = \sum_j W_{2,ij} h_j + b_{2,i}$.Index form shows which entry of $z_2$ each entry of $W_2$ and $b_2$ builds.
2. $W_{2,ij}$ appears only in $z_{2,i}$, with coefficient $h_j$.It sits in row $i$ of $W_2$, and row $i$ builds only $z_{2,i}$; within that sum, only term $j$ contains it.
3. $\partial L/\partial W_{2,ij} = \sum_{i'} \dfrac{\partial L}{\partial z_{2,i'}}\,\dfrac{\partial z_{2,i'}}{\partial W_{2,ij}} = \delta_{2,i}\, h_j$.$L$ depends on $W_2$ only through $z_2$; by step 2 only the $i' = i$ term is nonzero.
4. $(\delta_2 h^\top)_{ij} = \delta_{2,i} h_j$, with shapes $(k \times 1)(1 \times m) = k \times m$.An outer product $ab^\top$ has entries $a_i b_j$: the row index comes from $\delta_2$ and the column index from $h$, matching step 3.
5. $\partial L/\partial b_{2,i} = \delta_{2,i}$.$b_{2,i}$ appears only in $z_{2,i}$, with coefficient $1$: $\partial z_2/\partial b_2 = I_k$ (the Jacobians page, Problem 2).
6. $\nabla_{W_2} L = \delta_2 h^\top$ ($k\times m$); $\nabla_{b_2} L = \delta_2$$k \times m$ and $k$, the shapes of $W_2$ and $b_2$ from Problem 1. It is the Jacobians page's $(Wx - y)\,x^\top$ with $\delta_2$ as the incoming gradient: the gradient at the output of the affine map times its input, transposed.

Problem 4

1. $\partial z_2/\partial h = W_2$, $k \times m$.$z_2 = W_2 h + b_2$ is affine in $h$ (the Jacobians page, Problem 2).
2. $\nabla_h L = (\partial z_2/\partial h)^\top \nabla_{z_2} L$.$L$ depends on $h$ only through $z_2$, so the chain rule for gradients applies.
3. $\nabla_h L = W_2^\top \delta_2$Shapes $(m \times k)(k \times 1) = m \times 1$, the shape of $h$. Entry $j$ is $\sum_i W_{2,ij}\,\delta_{2,i}$: hidden unit $j$ collects the output deltas weighted by its outgoing weights, column $j$ of $W_2$.

Problem 5

1. $\partial h/\partial z_1 = \operatorname{diag}(\sigma'(z_1))$, $m \times m$.$h = \sigma(z_1)$ is elementwise (the Jacobians page, Problem 1).
2. $\delta_1 = (\partial h/\partial z_1)^\top \nabla_h L = \operatorname{diag}(\sigma'(z_1))\,W_2^\top\delta_2$.$L$ depends on $z_1$ only through $h$; a diagonal matrix is its own transpose; $\nabla_h L$ is Problem 4.
3. $\operatorname{diag}(\sigma'(z_1))\,v = \sigma'(z_1) \odot v$ for any $v \in \mathbb{R}^m$.A diagonal matrix acting on a vector scales entry $i$ by diagonal entry $i$: the vector-Jacobian product of the Jacobians page, Problem 6, so the $m \times m$ matrix is never built.
4. $\delta_1 = \sigma'(z_1)\odot(W_2^\top\delta_2)$$(m \times 1) \odot (m \times 1) = m \times 1$, the shape of $z_1$. For the sigmoid, $\sigma'(z_1) = h \odot (1 - h)$, computed from the $h$ saved in the forward pass.

Problem 6

1. $z_1 = W_1 x + b_1$ has the form of $z_2 = W_2 h + b_2$, with $x$ in place of $h$ and $\delta_1 = \nabla_{z_1} L$ in place of $\delta_2$.Problem 3 used only that form and the gradient at its output, so its argument applies unchanged.
2. $\partial L/\partial W_{1,ij} = \delta_{1,i}\, x_j$ and $\partial L/\partial b_{1,i} = \delta_{1,i}$.$W_{1,ij}$ appears only in $z_{1,i}$, with coefficient $x_j$, and $b_{1,i}$ only in $z_{1,i}$, with coefficient $1$ (Problem 3, steps 2 to 5).
3. $\nabla_{W_1} L = \delta_1 x^\top$ ($m\times d$); $\nabla_{b_1} L = \delta_1$$(m \times 1)(1 \times d) = m \times d$ and $m$, the shapes of $W_1$ and $b_1$ from Problem 1.

Problem 7

1. $\partial z_1/\partial x = W_1$, $m \times d$.$z_1 = W_1 x + b_1$ is affine in $x$.
2. $\nabla_x L = (\partial z_1/\partial x)^\top \delta_1 = W_1^\top\delta_1$, with shapes $(d \times m)(m \times 1) = d \times 1$.$L$ depends on $x$ only through $z_1$; the result has the shape of $x$.
3. No parameter update uses it.$x$ is data, not a parameter, so gradient descent never changes it.
4. $\nabla_x L = W_1^\top\delta_1$You want it for three things. Adversarial examples: a small step along $\operatorname{sign}(\nabla_x L)$ raises the loss the most, to first order, among changes of the same maximum entry size. Saliency: the size of entry $j$ says how sensitive the loss is to input feature $j$. And if $x$ is itself the output of a layer below, $\nabla_x L$ is that layer's incoming gradient, from which it gets its own $\delta_0$ exactly as Problem 5 got $\delta_1$ from $W_2^\top\delta_2$.

Problem 8

Write $x^{(n)}$ for example $n$ as a column, so row $n$ of $X$ is $x^{(n)\top}$, and likewise $h^{(n)}$, $\hat y^{(n)}$, $y^{(n)}$, and $\delta_2^{(n)} = \hat y^{(n)} - y^{(n)}$, $\delta_1^{(n)}$ for the single-example deltas of Problems 2 and 5.

1. Row $n$ of $Z_1$ is $x^{(n)\top} W_1^\top + b_1^\top = (W_1 x^{(n)} + b_1)^\top$, and likewise down the network.Row $n$ of $XW_1^\top$ is row $n$ of $X$ times $W_1^\top$, and $\mathbf{1}b_1^\top$ adds $b_1^\top$ to every row. So the batch is $N$ copies of the single-example network, one per row, sharing the parameters.
2. $L_n$ depends only on row $n$ of $X$, $Z_1$, $H$, $Z_2$ and $\hat Y$, so row $n$ of $\Delta_2$ is $\tfrac1N\,\delta_2^{(n)\top}$.Only the $n$-th term of the mean contains row $n$ of $Z_2$, and it carries the factor $\tfrac1N$; the rest is Problem 2 for example $n$ (the softmax page, Problem 7).
3. $\Delta_2 = \tfrac1N(\hat Y - Y)$, $N \times k$.Stack the rows of step 2.
4. $\nabla_{W_2} L = \sum_n \tfrac1N\,\delta_2^{(n)} h^{(n)\top}$.$W_2$ is used by every row, so the loss reaches it through all $N$ copies and its gradient is the sum of their contributions; each is Problem 3 for example $n$, scaled by $\tfrac1N$.
5. For matrices $A$ and $B$ with rows $a^{(n)\top}$ and $b^{(n)\top}$, $\sum_n a^{(n)} b^{(n)\top} = A^\top B$.$(A^\top B)_{ij} = \sum_n A_{ni} B_{nj} = \sum_n a^{(n)}_i b^{(n)}_j$, the $(i,j)$ entry of the sum of outer products.
6. $\nabla_{W_2} L = \Delta_2^\top H$, with shapes $(k \times N)(N \times m) = k \times m$.Step 5 with $A = \Delta_2$, whose rows are $\tfrac1N\delta_2^{(n)\top}$, and $B = H$. The shape is that of $W_2$.
7. $\nabla_{b_2} L = \sum_n \tfrac1N\,\delta_2^{(n)} = \Delta_2^\top\mathbf{1}$, with shapes $(k \times N)(N \times 1) = k \times 1$.$b_2$ is added to every row, so its gradient is the sum over rows of Problem 3's $\delta_2$; $\Delta_2^\top\mathbf{1}$ sums the rows of $\Delta_2$.
8. $\nabla_H L = \Delta_2 W_2$, with shapes $(N \times k)(k \times m) = N \times m$.Row $n$ is Problem 4 for example $n$, transposed: $(W_2^\top\,\tfrac1N\delta_2^{(n)})^\top = \tfrac1N\delta_2^{(n)\top} W_2$.
9. $\Delta_1 = \sigma'(Z_1) \odot (\Delta_2 W_2)$, $N \times m$, with $\sigma'$ applied to every entry of $Z_1$.Problem 5 row by row: an elementwise product is taken entry by entry, so it does not care whether the entries are laid out as columns or rows. Row $n$ is $\tfrac1N\delta_1^{(n)\top}$, Problem 5 for example $n$.
10. $\nabla_{W_1} L = \Delta_1^\top X$, $(m \times N)(N \times d) = m \times d$; $\nabla_{b_1} L = \Delta_1^\top\mathbf{1}$, $(m \times N)(N \times 1) = m \times 1$.Steps 4 to 7 one layer down, with $\Delta_1$ in place of $\Delta_2$ and $X$ in place of $H$.
11. $\nabla_X L = \Delta_1 W_1$, $(N \times m)(m \times d) = N \times d$.Step 8 one layer down: row $n$ is $\tfrac1N(W_1^\top\delta_1^{(n)})^\top$, Problem 7 for example $n$ scaled by $\tfrac1N$.
12. $\Delta_2 = \tfrac1N(\hat Y - Y)$ ($N\times k$); $\nabla_{W_2}L = \Delta_2^\top H$ ($k\times m$); $\nabla_{b_2}L = \Delta_2^\top\mathbf{1}$; $\Delta_1 = \sigma'(Z_1)\odot(\Delta_2 W_2)$ ($N\times m$); $\nabla_{W_1}L = \Delta_1^\top X$; $\nabla_{b_1}L = \Delta_1^\top\mathbf{1}$; $\nabla_X L = \Delta_1 W_1$ ($N\times d$)Every gradient has the shape of its variable. The $\tfrac1N$ enters once, in $\Delta_2$, and everything after it is linear in $\Delta_2$, so it is carried through and never applied again.

Problem 9

1. $\operatorname{ReLU}'(t) = 1$ for $t > 0$ and $0$ for $t < 0$; write $[z_1 > 0]$ for the vector whose entry $i$ is $1$ if $z_{1,i} > 0$ and $0$ otherwise.ReLU is $t$ to the right of $0$ and $0$ to the left. At $t = 0$ the one-sided slopes are $0$ and $1$, so the derivative is undefined; it is taken as $0$, and a pre-activation exactly at $0$ is rare with real-valued data.
2. $\delta_2$, $\nabla_{W_2} L$, $\nabla_{b_2} L$ and $\nabla_h L$ keep their formulas, with $h = \operatorname{ReLU}(z_1)$.Problems 2 to 4 used $h$ only as a vector feeding $z_2$; none of them differentiated the activation. The values change, the formulas do not.
3. $\delta_1 = \operatorname{ReLU}'(z_1) \odot (W_2^\top\delta_2)$.Problem 5 used only that the activation is elementwise, so its derivative $\sigma'(z_1)$ is replaced by $\operatorname{ReLU}'(z_1)$, which is $[z_1 > 0]$ by step 1.
4. $\nabla_{W_1} L = \delta_1 x^\top$, $\nabla_{b_1} L = \delta_1$, $\nabla_x L = W_1^\top\delta_1$.Problems 6 and 7 start from $\delta_1$ and never look at the activation; the new $\delta_1$ flows into them.
5. $\delta_1 = [z_1 > 0]\odot(W_2^\top\delta_2)$; $\nabla_x L = W_1^\top\delta_1$; nothing else changesOnly the elementwise factor in Problem 5 changes. A hidden unit with $z_{1,i} \le 0$ passes back nothing: entry $i$ of $\delta_1$ is $0$, so row $i$ of $\nabla_{W_1} L$ and entry $i$ of $\nabla_{b_1} L$ are zero for this example. A unit that is negative on every input never updates, which is a dead ReLU.

Problem 10

1. $\tfrac\lambda2\|W_2\|_F^2 = \tfrac\lambda2\sum_{i,j} W_{2,ij}^2$.The squared Frobenius norm is the sum of the squares of the entries.
2. $\partial\big(\tfrac\lambda2\|W_2\|_F^2\big)/\partial W_{2,ij} = \lambda W_{2,ij}$, so the penalty's gradient is $\lambda W_2$, $k \times m$.Only one term of the sum contains $W_{2,ij}$, and $\tfrac\lambda2 t^2$ has derivative $\lambda t$. This is the matrix-calculus page's $\nabla_X\|X\|_F^2 = 2X$, times $\tfrac\lambda2$.
3. $\nabla_{W_2} L = \delta_2 h^\top$, $k \times m$.Problem 3; the penalty does not involve $z_2$, so $\delta_2$ is unchanged.
4. $\nabla_{W_2} L' = \delta_2 h^\top + \lambda W_2$The gradient of a sum is the sum of the gradients, and both terms are $k \times m$. A gradient step becomes $W_2 \leftarrow (1 - \eta\lambda)W_2 - \eta\,\delta_2 h^\top$: the update shrinks the weights by $1 - \eta\lambda$ and adds the data term, which is why this is called weight decay. The other gradients are unchanged, because the penalty contains only $W_2$.