EulerPad

Guides / Matrix calculus

Regression gradients: (2/N)Xᵀ(Xw − y), Xᵀ(σ(Xw) − y)/N and Xᵀ(S − Y)/N, with the 1/N kept in view

Linear, logistic and softmax regression are one-layer networks, and all three gradients have the same shape: the data matrix transposed, times the prediction error, divided by the number of examples. Least squares adds a 2 from the square; logistic and softmax regression don't, because the sigmoid's derivative cancels against the cross-entropy's. The conventions are those of the matrix calculus conventions guide: rows of X are examples, and every gradient has the shape of its variable. The mistakes come from dropping the 2/N, from copying the ridge formula of the summed loss onto the mean loss, and from multiplying the logistic error by an extra σ′.

Short answer

X is N × d, one example per row. Least squares, L = (1/N)‖Xw − y‖2: ∇wL = (2/N)XT(Xw − y), minimized where XTXw = XTy. Ridge, adding λ‖w‖2: w* = (XTX + Nλ I)−1XTy, with an N on λ. Logistic, mean cross-entropy with p = σ(Xw): ∇wL = (1/N)XT(p − y). Softmax, Z = XW + 1bT with W of shape d × k: ∇WL = (1/N)XT(S − Y) and ∇bL = (1/N)(S − Y)T1.

Why it works

Each loss reaches w through the predictions, so the chain rule has two links: the gradient at the predictions, then the transposed Jacobian of Xw, which is XT. For the mean squared error the gradient at the residual r = Xw − y is (2/N)r, giving (2/N)XTr. Entry j is a weighted average over examples: each residual times that example's feature j.

For cross-entropy on a sigmoid, the derivative of the loss in p is (p − y)/(p(1 − p)), and the sigmoid contributes σ′ = p(1 − p). They cancel, so the gradient at the logit is just the error p − y. Softmax with cross-entropy does the same thing one class at a time, with S − Y in place of p − y. The 1/N of the mean rides along unchanged, and it never moves the minimizer, only the size of each gradient step.

Example 1: the least-squares gradient on three points

Fit y ≈ w1 + w2x to the points (0, 1), (1, 2), (2, 4). Find L and ∇wL at w = (0, 1).

  1. X = 101112A column of ones for the intercept, then the feature. N = 3, d = 2.
  2. Xw = (0, 1, 2)Each row times w.
  3. r = Xw − y = (−1, −1, −2)The residuals, with y = (1, 2, 4).
  4. L = (1 + 1 + 4)/3 = 2The mean squared residual.
  5. XTr = (−4, −5)The sum of the residuals, then the sum weighted by x: 0 − 1 − 4.
  6. ∇wL = (2/3)(−4, −5)Shape 2 × 1, like w.
  7. ≈ (−2.667, −3.333)Both entries are negative, so both weights should grow. A central finite difference of L gives the same numbers.

Example 2: the normal equations, with and without ridge

For the data of Example 1, find the least-squares w*, then the minimizer of L + λ‖w‖2 with λ = 1/3.

  1. XTX = 3335Entries: N, Σx, Σx, Σx2.
  2. XTy = (7, 10)Σy and Σxy.
  3. XTXw = XTyThe gradient set to zero, times N/2.
  4. w* = (5/6, 3/2)Solve the 2 × 2 system; the determinant is 6. The gradient there is 0, and the loss drops from 2 to 1/18.
  5. (XTX + Nλ I)w = XTyRidge. Clearing the 1/N of the data term puts an N on the penalty.
  6. Nλ = 1So add 1 to each diagonal entry: 4336.
  7. wλ = (4/5, 19/15) ≈ (0.8, 1.267)The determinant is 15. Both weights shrink toward 0, and the ridge loss's finite-difference gradient there is 0.

Example 3: the logistic gradient on two examples

Two examples, X = 121−1 with labels y = (1, 0), and w = (0, 0.5). Find ∇wL for the mean cross-entropy.

  1. Xw = (1, −0.5)The logits.
  2. p = σ(Xw)The predicted probabilities of class 1 …
  3. ≈ (0.7311, 0.3775)… which are σ(1) and σ(−0.5).
  4. p − y ≈ (−0.2689, 0.3775)The errors: the first example is under-predicted, the second over-predicted.
  5. XT(p − y) ≈ (0.1086, −0.9154)Column sums weighted by the errors: −0.2689 + 0.3775, and 2(−0.2689) − 0.3775.
  6. ∇wL ≈ (0.0543, −0.4577)Divide by N = 2. No σ′ factor appears. A finite difference of the mean cross-entropy (L ≈ 0.3937) agrees.

Example 4: softmax regression, shapes first

Three classes, X = 1210, the first example in class 2 and the second in class 1, starting from W = 0 (2 × 3) and b = 0. Find ∇WL and ∇bL.

  1. S = all 1/3Every logit is 0, so every class gets the same probability.
  2. S − Y = 1/3−2/31/3−2/31/31/3Subtract 1 at each example's class. Each row sums to 0.
  3. (d × N)(N × k) = 2 × 3XT(S − Y) has the shape of W, which is how to tell it from (S − Y)TX.
  4. ∇WL = (1/2)XT(S − Y)Row j weights the rows of S − Y by feature j.
  5. = −1/6−1/61/31/3−2/31/3Row 1 averages the two rows of S − Y; row 2 is its first row, times 2, over 2.
  6. ∇bL = (−1/6, −1/6, 1/3)The column means of S − Y: classes 1 and 2 each appear once and are pushed up, class 3 never appears and is pushed down. Finite differences agree with both gradients.

Common mistakes, and the exact line they happen on

1. Dropping the 2/N

The gradient of Example 1.

  1. XTr = (−4, −5)Right so far.
  2. ∇wL = (−4, −5)This is the mistake. XT(Xw − y) is the gradient of ½‖Xw − y‖2, not of the mean squared error. Setting either to zero gives the same w*, which hides the slip, but every step is N/2 = 1.5 times too large here, and the factor changes with the batch size. The gradient is (−2.667, −3.333).

2. The ridge formula without the N

The ridge fit of Example 2.

  1. L + λ‖w‖2, λ = 1/3Right so far: a mean data term plus a penalty.
  2. w = (XTX + λI)−1XTyThis is the mistake. The textbook formula belongs to the summed loss ‖Xw − y‖2 + λ‖w‖2. Here it gives (66/79, 111/79) ≈ (0.835, 1.405), where the gradient of the loss actually written down is (0.371, 0.624), not 0. With Nλ in place of λ the answer is (0.8, 1.267).

3. An extra p(1 − p) in the logistic gradient

The gradient of Example 3.

  1. ∂pn/∂tn = pn(1 − pn)Right so far: the sigmoid's derivative at each logit.
  2. (1/N)XT[(p − y) ⊙ p ⊙ (1 − p)]This is the mistake. p − y is already the gradient at the logit: the cross-entropy's 1/(p(1 − p)) cancelled the σ′. The extra factor gives (0.0179, −0.0972), a third and a fifth of the true (0.0543, −0.4577), and it vanishes for an example that is confidently wrong, exactly where the step should be largest.

Practice

  1. For one example x = (1, 3), y = 2 and the loss (xTw − y)2, find the gradient at w = (1, 0).

    Show answer

    2(xTw − y)x = 2(1 − 2)(1, 3) = (−2, −6).

  2. A logistic model gives p = 0.8 on an example with y = 1 and x = (1, −2). Find that example's gradient.

    Show answer

    (p − y)x = −0.2(1, −2) = (−0.2, 0.4).

  3. Softmax regression on 100 examples with 5 features and 10 classes. What are the shapes of ∇WL and ∇bL?

    Show answer

    5 × 10 and 10, the shapes of W and b. N = 100 doesn't appear in either shape.

  4. Switching from the mean squared error to the summed squared error over 50 examples, what happens to the gradient and to w*?

    Show answer

    The gradient is 50 times larger; w* doesn't move, because scaling a loss by a positive constant keeps its minimizer.

  5. Find the Hessian of the mean squared error of Example 1.

    Show answer

    (2/N)XTX = 22210/3, the same at every w.

Practice this

Printable PDF: regression-gradients.pdf

Related guides

All guides