Guides / Matrix calculus

Activation function derivatives: σ(1 − σ), 1 − tanh², the ReLU mask, and which ones you can compute from the output

An activation acts on each entry separately, so its Jacobian is diagonal and the backward pass through it is one elementwise product: dz = dh ⊙ f′(z). All the work is in f′. The sigmoid, tanh and ReLU have derivatives that can be computed from the saved output h, which is why frameworks keep h and throw z away; SiLU and GELU don't, so their backward pass needs z. The conventions are those of the matrix calculus conventions guide. The mistakes come from feeding the pre-activation into an output formula, from using the ReLU's output as its mask, and from treating SiLU's sigmoid as a constant.

Why it works

Since h_i depends only on z_i, the Jacobian ∂h/∂z is diag(f′(z)). The chain rule for a scalar loss multiplies dh by the transposed Jacobian, and a diagonal matrix times a vector scales each entry, so no matrix is ever built.

A derivative can be written in terms of h exactly when each output value comes from inputs with a single slope. The sigmoid and tanh are increasing and invertible, so h pins down z and hence the slope; the ReLU's slope depends only on the sign of z, and h has the same sign. SiLU and GELU dip below zero and come back up, so one output value is reached from two inputs with different slopes, and no formula in h alone can be right for both.

Example 1: sigmoid and tanh from their outputs

A forward pass saved σ(1) and tanh 1. Find both derivatives at z = 1 from the saved values.

1. h = σ(1) ≈ 0.7311The saved sigmoid output.
2. σ′(1) = h(1 − h) ≈ 0.1966No exponential needed in the backward pass.
3. h = tanh 1 ≈ 0.7616The saved tanh output.
4. tanh′(1) = 1 − h² ≈ 0.4200Again from the output alone.
5. 0.4200 = 4σ′(2)Because tanh z = 2σ(2z) − 1, tanh's slope is four times the sigmoid's at twice the input: 4 × 0.1050. Finite differences of σ and tanh at 1 agree to every digit shown.

Example 2: a backward pass through ReLU and leaky ReLU

Pre-activations z = (1.5, −0.5, −2, 2) and incoming gradient dh = (0.3, −1, 2, −0.4). Find dz for a ReLU, then for a leaky ReLU with α = 0.1.

1. 1[z > 0] = (1, 0, 0, 1)The ReLU's derivative is a mask: 1 where the unit is on.
2. dz = dh ⊙ maskPass the active entries through unchanged …
3. = (0.3, 0, 0, −0.4)… and block the rest, however large their dh.
4. (1, 0.1, 0.1, 1)The leaky ReLU's slopes: α on the negative side instead of 0.
5. dz = (0.3, −0.1, 0.2, −0.4)The off units now get a tenth of their gradient, so they can still learn their way back. A finite difference of dh^Tf(z) in each z_i gives both answers.

Example 3: SiLU and GELU at z = 1

Find the derivatives of SiLU(z) = zσ(z) and GELU(z) = zΦ(z) at z = 1.

1. SiLU′ = σ + zσ′Product rule: the gate σ(z) depends on z too.
2. = σ(z)(1 + z(1 − σ(z)))Factor out σ(z), using σ′ = σ(1 − σ).
3. ≈ 0.7311 × 1.2689At z = 1.
4. ≈ 0.9277More than the gate's 0.7311.
5. GELU′ = Φ(z) + zφ(z)Product rule, with Φ′ = φ, the normal density.
6. ≈ 0.8413 + 0.2420 ≈ 1.0833Greater than 1, so a GELU layer can enlarge a gradient, which a sigmoid or tanh layer never does. Both derivatives match central finite differences.

Example 4: cross-entropy from a logit of 40

A binary classifier outputs the logit t = 40 for an example whose label is y = 0. Find the loss and its gradient in 64-bit floating point.

1. ℓ = −log(1 − σ(40))The definition, for y = 0.
2. σ(40) rounds to 1It is within e⁻⁴⁰ ≈ 4 × 10⁻¹⁸ of 1, closer than 64-bit floats can resolve. This happens from about t = 37, and from t = 17 in 32-bit.
3. −log 0 = ∞Computed this way, the loss is infinite and its gradient NaN.
4. ℓ = max(t, 0) − ytThe same function, rearranged …
5. + log(1 + e^−|t|)… so the exponential never exceeds 1.
6. = 40 − 0 + log(1 + e⁻⁴⁰) ≈ 40Finite and correct.
7. dℓ/dt = σ(40) − 0 ≈ 1The full-sized gradient a confidently wrong prediction deserves. This is what binary_cross_entropy_with_logits computes.

Common mistakes, and the exact line they happen on

1. Feeding z into the output formula

A sigmoid unit with pre-activation z = 2.

1. dz = dh · σ′(z)Right so far.
2. σ′ = z(1 − z) = −2This is the mistake. The formula h(1 − h) needs the output h = σ(2) ≈ 0.8808, not the input. The right factor is σ′(2) ≈ 0.1050; the wrong one has the opposite sign and is 19 times larger. It only looks right while z happens to lie between 0 and 1.

2. Multiplying by the ReLU's output instead of its mask

The ReLU layer of Example 2.

1. h = (1.5, 0, 0, 2)Right so far: zero exactly where the gradient should be blocked.
2. dz = dh ⊙ h = (0.45, 0, 0, −0.8)This is the mistake. h has the right zeros, but on the active units it scales the gradient by the unit's own value instead of passing it through. That is the backward pass of ½ReLU(z)². The mask is 1[h > 0], giving (0.3, 0, 0, −0.4).

3. Treating SiLU's gate as a constant

The SiLU derivative of Example 3.

1. SiLU(z) = z · σ(z)Right so far: the input times a sigmoid gate.
2. SiLU′(1) = σ(1) ≈ 0.7311This is the mistake. The gate is computed from the same z, so the product rule adds zσ′(z) ≈ 0.1966. The derivative is 0.9277. At negative z the missing term changes the sign: SiLU′(−2) ≈ −0.091, while σ(−2) ≈ 0.12 is positive.

Practice

1. A tanh unit saved the output h = 0.6. Find tanh′(z).

   Show answer

   1 − 0.36 = 0.64.

2. A sigmoid unit saved h = 0.9, and dh = 2. Find dz.

   Show answer

   2 × 0.9 × 0.1 = 0.18.

3. A softplus unit saved h = ln 3. Find its derivative.

   Show answer

   1 − e^−h = 2/3. Check: e^z = 2, and σ(ln 2) = 2/3.

4. An ELU with α = 1 saved h = −0.5. Find its derivative.

   Show answer

   h + α = 0.5, which is e^z at z = ln 0.5.

5. A gradient passes back through three sigmoid layers, with the weights left out. What is the most it can keep of its size?

   Show answer

   (1/4)³ = 1/64, since σ′ ≤ 1/4 at every unit.