Activation functions and their derivatives

Worked solutions

Problem 1

1. $\zeta'(t) = \dfrac{e^t}{1 + e^t}$.Chain rule on $\log u$ with $u = 1 + e^t$: the derivative is $u'/u$.
2. $\dfrac{e^t}{1 + e^t} = \dfrac{1}{e^{-t} + 1} = \sigma(t)$.Divide numerator and denominator by $e^t$.
3. $\log\sigma(t) = -\log(1 + e^{-t}) = -\zeta(-t)$.The log of a reciprocal is minus the log, and $\zeta(-t) = \log(1 + e^{-t})$ by definition.
4. $\dfrac{d}{dt}\log\sigma(t) = \dfrac{d}{dt}\big[-\zeta(-t)\big] = \zeta'(-t) = \sigma(-t) = 1 - \sigma(t)$.The chain rule through $-t$ brings a factor $-1$ that cancels the minus sign; then step 2 at $-t$, and the identity from the regression page.
5. $\zeta'(t) = \sigma(t)$; $\log\sigma(t) = -\zeta(-t)$; $(\log\sigma)'(t) = 1 - \sigma(t) = \sigma(-t)$The sigmoid is the slope of softplus, and the log-sigmoid is a negated softplus. Both identities return in Problem 10, where they make the cross-entropy loss safe to compute for large logits. The slope of $\log\sigma$ lies in $(0, 1)$: near $1$ for very negative $t$, where $\log\sigma(t) \approx t$, and near $0$ for large $t$, where $\sigma(t) \approx 1$.

Problem 2

1. $\tanh t = s/c$ with $s = \sinh t = \tfrac12(e^t - e^{-t})$ and $c = \cosh t = \tfrac12(e^t + e^{-t})$, and $s' = c$, $c' = s$.Differentiate the exponentials term by term; the minus sign in $s$ turns into a plus and back.
2. $\tanh' = \dfrac{c\cdot c - s\cdot s}{c^2} = 1 - \dfrac{s^2}{c^2} = 1 - \tanh^2$.Quotient rule, then split the fraction.
3. $2\sigma(2t) - 1 = \dfrac{2}{1 + e^{-2t}} - 1 = \dfrac{1 - e^{-2t}}{1 + e^{-2t}}$.Put both terms over the common denominator.
4. $\dfrac{1 - e^{-2t}}{1 + e^{-2t}} = \dfrac{e^t - e^{-t}}{e^t + e^{-t}} = \tanh t$.Multiply numerator and denominator by $e^t$, using $e^t e^{-2t} = e^{-t}$.
5. $\tanh'(t) = \dfrac{d}{dt}\big[2\sigma(2t) - 1\big] = 2\cdot 2\,\sigma'(2t) = 4\sigma'(2t)$.Chain rule through the inner $2t$.
6. $\tanh'(t) = 1 - \tanh^2(t) = 4\sigma'(2t)$; $\tanh' \in (0, 1]$ with its maximum $1$ at $t = 0$, and $\sigma' \in (0, \tfrac14]$ with its maximum at $t = 0$$\tanh$ is the sigmoid stretched vertically to $(-1, 1)$ and compressed horizontally by $2$, so its slope is four times the sigmoid's at the matching point. Both derivatives are written in terms of the output in Problem 7, and the bounds are what Problem 8 uses: a backward step through either activation can shrink a gradient but never grow it.

Problem 3

1. For $t > 0$, $\operatorname{relu}(t) = t$ and $\operatorname{relu}'(t) = 1$; for $t < 0$, $\operatorname{relu}(t) = 0$ and $\operatorname{relu}'(t) = 0$.On each side of $0$ the function is a straight line, with slopes $1$ and $0$.
2. $f_\alpha'(t) = 1$ for $t > 0$ and $\alpha$ for $t < 0$.The same, with slope $\alpha$ on the left.
3. At $t = 0$ the one-sided slopes differ, so neither function is differentiable there; for the ReLU every value in $[0, 1]$ is a subgradient, and frameworks use $\operatorname{relu}'(0) = 0$.A convex function's subgradients at a kink are the slopes between the two one-sided slopes; for a continuous pre-activation $P(t = 0) = 0$, so the choice has no effect on training.
4. For $c > 0$, $\operatorname{relu}(ct) = \max(ct, 0) = c\max(t, 0) = c\operatorname{relu}(t)$; differentiating in $t$ gives $c\operatorname{relu}'(ct) = c\operatorname{relu}'(t)$, so $\operatorname{relu}'(ct) = \operatorname{relu}'(t)$.A positive factor comes out of a maximum; the chain rule on the left and the constant $c$ on the right.
5. For the ReLU, $h > 0$ exactly when $t > 0$, so $\operatorname{relu}'(t) = \mathbb{1}[h > 0]$; for the leaky ReLU, $h$ has the sign of $t$ because $\alpha > 0$, so $f_\alpha'(t) = \mathbb{1}[h > 0] + \alpha\mathbb{1}[h < 0]$.Each derivative depends on $t$ only through its sign, and the output carries the sign.
6. $\operatorname{relu}'(t) = \mathbb{1}[t > 0] = \mathbb{1}[h > 0]$ and $f_\alpha'(t) = \mathbb{1}[t > 0] + \alpha\mathbb{1}[t < 0] = \mathbb{1}[h > 0] + \alpha\mathbb{1}[h < 0]$, with the value at $0$ a convention; $\operatorname{relu}(ct) = c\operatorname{relu}(t)$ for $c > 0$, so the derivative is constant along each rayThe derivative is a mask, so a ReLU backward pass needs only one bit per unit. Positive homogeneity is why a ReLU network's output is piecewise linear in its input and why the scale of one layer's weights can be moved into the next layer's. A unit whose pre-activation is negative for every input in the data has mask $0$ everywhere, receives no gradient, and never recovers: the dead ReLU that the leaky slope $\alpha$ exists to prevent.

Problem 4

1. $f'(t) = \sigma(t) + t\,\sigma'(t)$.Product rule.
2. $f'(t) = \sigma(t)\big(1 + t(1 - \sigma(t))\big)$.$\sigma' = \sigma(1 - \sigma)$ from the regression page's Problem 4, then factor out $\sigma(t)$.
3. $t\,\sigma(t)(1 - \sigma(t)) = h(1 - \sigma(t))$, so $f'(t) = \sigma(t) + h - h\sigma(t) = h + \sigma(t)(1 - h)$.Replace $t\,\sigma(t)$ by $h$ in the second term of step 1 and collect.
4. At $t = -2$: $\sigma(-2) = 1/(1 + e^2) \approx 0.1192$, so $1 - \sigma(-2) \approx 0.8808$ and $f'(-2) \approx 0.1192\,(1 - 2\cdot 0.8808) = 0.1192\cdot(-0.7616) \approx -0.0908$.Step 2 with $e^2 \approx 7.389$.
5. $f'(t) = \sigma(t)\big(1 + t(1 - \sigma(t))\big) = h + \sigma(t)(1 - h)$; $f'(-2) \approx -0.091 < 0$, so SiLU decreases thereSiLU has a minimum of about $-0.278$ near $t \approx -1.28$ and rises back towards $0$ as $t \to -\infty$, so its slope is negative on the far left. For positive $t$ the derivative exceeds $1$ (at $t = 2$ it is about $1.09$), so unlike the sigmoid and $\tanh$ it can enlarge a gradient. The output form still contains $\sigma(t)$, which is not a function of $h$ alone; Problem 7 says why.

Problem 5

1. $\operatorname{GELU}'(t) = \Phi(t) + t\,\Phi'(t) = \Phi(t) + t\,\varphi(t)$.Product rule; the derivative of a distribution function is its density.
2. $g'(t) = \tfrac12\big(1 + \tanh u\big) + \tfrac12 t\,(1 - \tanh^2 u)\,u'(t)$.Product rule on $t$ times the bracket, then the chain rule through $u$, with $\tanh' = 1 - \tanh^2$ from Problem 2.
3. $u'(t) = \sqrt{2/\pi}\,(1 + 3\cdot 0.044715\,t^2) = \sqrt{2/\pi}\,(1 + 0.134145\,t^2)$.Power rule on the cubic.
4. $\operatorname{GELU}'(t) = \Phi(t) + t\,\varphi(t)$ with $\varphi(t) = e^{-t^2/2}/\sqrt{2\pi}$; $g'(t) = \tfrac12(1 + \tanh u) + \tfrac12 t\,(1 - \tanh^2 u)\sqrt{2/\pi}\,(1 + 0.134145\,t^2)$Both give $\tfrac12$ at $t = 0$, since $\Phi(0) = \tfrac12$ and $\tanh 0 = 0$. $\operatorname{GELU}'$ tends to $1$ as $t \to \infty$ and to $0$ as $t \to -\infty$, and like SiLU it is negative on part of the left: at $t = -1.5$, $\Phi(-1.5) \approx 0.067$ while $1.5\,\varphi(1.5) \approx 0.194$, so the derivative is about $-0.13$. The approximation tracks the exact function to within a few thousandths on $[-3, 3]$, but its derivative needs the inner factor $u'$ (Mistake 5).

Problem 6

1. For $t \ge 0$: $\zeta(t) = \log\big(e^t(1 + e^{-t})\big) = t + \log(1 + e^{-t}) = \operatorname{relu}(t) + \log(1 + e^{-|t|})$.Factor $e^t$ out of $1 + e^t$ inside the log; the log of a product is a sum; $|t| = t$ here.
2. For $t < 0$: $\zeta(t) = \log(1 + e^t) = \operatorname{relu}(t) + \log(1 + e^{-|t|})$.$\operatorname{relu}(t) = 0$ and $e^t = e^{-|t|}$.
3. $0 < \log(1 + e^{-|t|}) \le \log 2$, with the maximum at $t = 0$ and the value tending to $0$ as $|t| \to \infty$.$e^{-|t|} \in (0, 1]$, and $\log(1 + u)$ is increasing in $u$ with $\log 1 = 0$.
4. $\zeta'' = \sigma' = \sigma(1 - \sigma) > 0$.Problem 1 gives $\zeta' = \sigma$, and $\sigma \in (0, 1)$ makes both factors positive.
5. $\zeta(t) = \max(t, 0) + \log(1 + e^{-|t|})$ with $0 < \zeta(t) - \operatorname{relu}(t) \le \log 2$; $\zeta''(t) = \sigma(t)(1 - \sigma(t)) > 0$, so softplus is convexSoftplus is the ReLU with its kink rounded off, never more than $\log 2 \approx 0.693$ above it. Computed as $\log(1 + e^t)$, it overflows in double precision once $t$ exceeds about $709$, and for $t$ near $-40$ the $e^t$ vanishes next to the $1$; in the stable form $e^{-|t|} \le 1$ never overflows and log1p keeps the small values exact. Convexity is the statement that the slope $\sigma$ only ever increases, from $0$ to $1$, where the ReLU's jumps.

Problem 7

1. Sigmoid: $\sigma'(z) = \sigma(z)(1 - \sigma(z)) = h(1 - h)$; $\tanh$: $1 - \tanh^2(z) = 1 - h^2$.The regression page's Problem 4 and Problem 2 here, with the output substituted.
2. ReLU: $\mathbb{1}[h > 0]$; leaky ReLU: $\mathbb{1}[h > 0] + \alpha\mathbb{1}[h < 0]$.Problem 3.
3. Softplus: $h = \log(1 + e^z)$ gives $e^h = 1 + e^z$, so $e^z = e^h - 1$ and $\zeta'(z) = \sigma(z) = \dfrac{e^z}{1 + e^z} = \dfrac{e^h - 1}{e^h} = 1 - e^{-h}$.Invert the softplus by exponentiating, then write $\sigma$ with $e^z$ in the numerator (Problem 1, step 2 read backwards) and substitute.
4. ELU: for $z \ge 0$, $h = z \ge 0$ and $f' = 1$; for $z < 0$, $h = \alpha(e^z - 1) < 0$ and $f' = \alpha e^z = h + \alpha$.Differentiate each branch; on the left, $\alpha e^z = \alpha(e^z - 1) + \alpha$. The sign of $h$ says which branch applies.
5. A formula in $h$ alone exists only when each output comes from inputs with the same slope. SiLU and GELU are not monotonic (Problems 4 and 5): SiLU takes the value $-0.2$ at $t \approx -0.55$, where its slope is about $0.24$, and again at $t \approx -2.4$, where its slope is about $-0.10$.One value of $h$, two different derivatives, so no function of $h$ can give both.
6. $\sigma' = h(1 - h)$; $\tanh' = 1 - h^2$; $\operatorname{relu}' = \mathbb{1}[h > 0]$; $f_\alpha' = \mathbb{1}[h > 0] + \alpha\mathbb{1}[h < 0]$; $\zeta' = 1 - e^{-h}$; $\operatorname{ELU}' = 1$ for $h \ge 0$ and $h + \alpha$ for $h < 0$; SiLU and GELU have no such formula, so their backward pass must keep $z$This is why a framework's sigmoid, $\tanh$ and ReLU save their output and free the input, while SiLU and GELU save the input. Each formula is checked numerically against finite differences of the forward function, which is the test to run on a hand-written backward pass as well.

Problem 8

1. $h_i = f(z_i)$ depends on $z_i$ alone, so $\partial h_i/\partial z_j = f'(z_i)$ for $j = i$ and $0$ otherwise: $\partial h/\partial z = \operatorname{diag}(f'(z))$.An elementwise map has a diagonal Jacobian, as on the Jacobians page.
2. $dz = (\partial h/\partial z)^\top dh = \operatorname{diag}(f'(z))\,dh = dh \odot f'(z)$.The chain rule for a scalar loss applies the transposed Jacobian to $dh$; a diagonal matrix is its own transpose, and applying it scales entry $i$ by $f'(z_i)$.
3. For $\tanh$, $f'(z) = 1 - h^2$, so $dz = dh \odot (1 - h^2)$.Problem 7.
4. $z_i = \sum_j W_{ij}x_j + b_i$, so $\partial L/\partial W_{ij} = dz_i\,x_j$ and $\partial L/\partial b_i = dz_i$: $\nabla_W L = dz\,x^\top$ and $\nabla_b L = dz$.$W_{ij}$ and $b_i$ appear in $z_i$ only, with coefficients $x_j$ and $1$; the outer product collects the entries in the shape of $W$.
5. $\|dz\|^2 = \sum_i dh_i^2\,f'(z_i)^2 \le \big(\max_i f'(z_i)^2\big)\|dh\|^2$.Each term is bounded by the largest factor times the same $dh_i^2$.
6. $dz = dh \odot f'(z)$; for $\tanh$, $\nabla_W L = \big(dh \odot (1 - h^2)\big)x^\top$ and $\nabla_b L = dh \odot (1 - h^2)$; $\|dz\| \le \|dh\|$ for $\tanh$ and $\|dz\| \le \tfrac14\|dh\|$ for the sigmoidProblem 2's bounds $\tanh' \le 1$ and $\sigma' \le \tfrac14$ in step 5. Through $L$ sigmoid layers the activations alone shrink a gradient by at least $4^{-L}$ unless the weights compensate, which is the vanishing-gradient problem the initialisation page quantifies; $\tanh$ at best preserves it, near $0$, and a ReLU passes each surviving entry unchanged and zeroes the rest. SiLU and GELU, whose derivatives exceed $1$ on part of the line, obey no such bound.

Problem 9

1. $\sigma(0) = \tfrac12$, $\sigma'(0) = \tfrac14$, and $\sigma'' = \sigma'(1 - 2\sigma)$, so $\sigma''(0) = 0$.Product rule on $\sigma' = \sigma(1 - \sigma)$: $\sigma'' = \sigma'(1 - \sigma) - \sigma\sigma'$.
2. $\tanh 0 = 0$, $\tanh'(0) = 1$, and $\tanh'' = -2\tanh\cdot\tanh'$, so $\tanh''(0) = 0$.Chain rule on $1 - \tanh^2$.
3. $\zeta(0) = \log 2$, $\zeta'(0) = \sigma(0) = \tfrac12$, $\zeta''(0) = \sigma'(0) = \tfrac14$.Problems 1 and 6.
4. SiLU: $f(0) = 0$, $f'(0) = \sigma(0) + 0 = \tfrac12$, and $f'' = 2\sigma' + t\sigma''$, so $f''(0) = 2\cdot\tfrac14 = \tfrac12$.Differentiate $f' = \sigma + t\sigma'$ once more with the product rule.
5. GELU: $f(0) = 0$, $f'(0) = \Phi(0) = \tfrac12$, and $f'' = 2\varphi + t\varphi'$, so $f''(0) = 2\varphi(0) = 2/\sqrt{2\pi} = \sqrt{2/\pi} \approx 0.798$.Differentiate $f' = \Phi + t\varphi$; $\varphi' = -t\varphi$ vanishes at $0$.
6. $\sigma(t) \approx \tfrac12 + \tfrac t4$; $\tanh t \approx t$; $\zeta(t) \approx \log 2 + \tfrac t2 + \tfrac{t^2}8$; $\operatorname{SiLU}(t) \approx \tfrac t2 + \tfrac{t^2}4$; $\operatorname{GELU}(t) \approx \tfrac t2 + \tfrac{t^2}{\sqrt{2\pi}}$The sigmoid and $\tanh$ are odd about their centre, so they have no $t^2$ term and are linear maps of gain $\tfrac14$ and $1$ near $0$ (the stretch of Problem 2). Softplus, SiLU and GELU all have gain $\tfrac12$ at $0$, and their $t^2$ terms are what bends them. The error of each approximation is of order $t^3$, as on the Taylor-series page; a network whose pre-activations stay small is close to a linear network, which is one reason initialisation scale matters.

Problem 10

1. $-\log\sigma(t) = \zeta(-t)$ and $-\log(1 - \sigma(t)) = -\log\sigma(-t) = \zeta(t)$.Problem 1 twice, with $1 - \sigma(t) = \sigma(-t)$.
2. $\ell = y\,\zeta(-t) + (1 - y)\,\zeta(t)$.Substitute step 1 into the definition.
3. $\zeta(-t) = \zeta(t) - t$.$\log(1 + e^{-t}) = \log\big(e^{-t}(e^t + 1)\big) = -t + \log(1 + e^t)$.
4. $\ell = y\big(\zeta(t) - t\big) + (1 - y)\zeta(t) = \zeta(t) - yt$.Collect the $\zeta(t)$ terms: $y + (1 - y) = 1$.
5. $\ell = \max(t, 0) - yt + \log(1 + e^{-|t|})$.Problem 6's stable form of $\zeta$.
6. $\dfrac{d\ell}{dt} = \zeta'(t) - y = \sigma(t) - y$.Problem 1.
7. $\ell = \zeta(t) - yt = \max(t, 0) - yt + \log(1 + e^{-|t|})$; $\dfrac{d\ell}{dt} = \sigma(t) - y$The definition evaluates $\sigma(t)$ first: at $t = -1000$ it underflows to $0$ and $\log 0$ is $-\infty$, so a confident wrong prediction gives an infinite loss and a nan gradient. The stable form gives $\ell = 1000$ exactly for $y = 1$ there, with gradient $\sigma(-1000) - 1 = -1$. This is PyTorch's binary_cross_entropy_with_logits and Keras's from_logits=True; the gradient $\sigma(t) - y$ is the regression page's prediction error, which the one-hidden-layer page takes as its starting $\delta$.