Ten problems on the backward pass of scaled dot-product attention: the gradients to V, A, the scores, Q and K, the projections in self-attention, the causal mask, why the scores are divided by √d, and the multi-head output projection, with worked solutions and the mistakes that transpose the wrong matrix.
Before you start
Scaled dot-product attention is three matrix products and a softmax: scores from queries and keys, weights from the scores, outputs from the weights and the values. Its backward pass is the same pieces run in reverse, and every step is either a matrix product, whose gradient is another matrix product with one factor transposed, or the softmax page's row result applied to each row. These ten problems derive each gradient, push them through the projections of self-attention, add the causal mask and the 1/d scaling, and finish with the multi-head output projection. The five mistakes are the ones that give a plausible shape or a plausible number: a missing transpose, a softmax Jacobian that mixes rows, a dropped ,1/d, a path through X left out, and a mask applied after the softmax.
The conventions are those of the previous pages: vectors are columns, Jacobians are in numerator layout, a gradient has the shape of the variable it is taken with respect to, and ⊙ is the elementwise product.
One head. ,Q∈Rn×d,K∈Rm×d and ,V∈Rm×dv, with rows as positions: n query positions, m key positions, width d for queries and keys and dv for values.
S=QK⊤/d is ;n×m;A=softmax(S) is taken row by row; O=AV is .n×dv.
L is a scalar loss that depends on ,Q,K and V only through .O. The upstream gradient is ,G=∇OL, and .G~:=∇AL.
Indices: i is a query position (a row of ,Q,,S,,A,),O),j a key position (a row of K and ,V, a column of S and ),A),l a column of Q and ,K, and c a column of V and .O.
1 is the all-ones vector of whatever length the product needs. rowsum(B):=B1 is a column holding the row sums of a matrix ,B, so rowsum(B)1⊤ copies each row's sum across that row.
The softmax page's row result: for one row, a=softmax(s) with upstream ,g~=∇aL, the Jacobian J=diag(a)−aa⊤ is symmetric (the softmax page, Problem 3), so ,∇sL=Jg~=a⊙g~−a(a⊤g~), that is, .∇s=a⊙(g~−(a⊤g~)1).
Every matrix below has rows as positions, so every formula is the row formula stacked.
Self-attention: ,Q=XWQ,,K=XWK,V=XWV with ,X∈Rn×dmodel, so .m=n.H is the number of heads and h indexes them.
Give the shapes of ,S,A and ,O, and the number of multiply-adds to form .S. What does that cost look like as the sequence length n=m grows?
··
Compute .∇VL.
··
Compute .G~=∇AL.
···
Compute ∇SL from G~ and ,A, using the softmax Jacobian row by row, and write it as one matrix expression with no m×m Jacobians.
··
Compute .∇QL.
··
Compute .∇KL.
···
Self-attention: ,Q=XWQ,,K=XWK,.V=XWV. Compute ,∇WQL,,∇WKL,∇WVL and .∇XL.
··
A causal mask sets Sij=−∞ for j>i before the softmax. What are the masked entries of ,A, and what are the masked entries of ?∇SL? Show it from Problem 4.
··
If the entries of q and k are independent with mean 0 and variance ,1, what is ?Var(q⊤k)? What does dividing by d achieve, and where does the factor appear in the backward pass?
···
Multi-head: O=[O1⋯OH]WO with Oh∈Rn×dv and .WO∈RHdv×dmodel. Compute ∇WOL and .∇OhL.
Answers
;S,A:n×m;;O:n×dv;S costs nmd multiply-adds, quadratic in the sequence length
masked ,Aij=0, and since ,∇SL=A⊙(⋯), masked (∇SL)ij=0 exactly; the unmasked entries follow Problem 4 with the masked columns contributing nothing to the row sums
;Var(q⊤k)=d; dividing by d makes the scores order 1 so the softmax does not saturate; the same d1 multiplies ∇QL and ∇KL (Problems 5–6) and nothing else
;∇WOL=[O1⋯OH]⊤G;∇OhL = columns (h−1)dv+1 to hdv of GWO⊤
Worked solutions
Problem 1
Give the shapes of ,S,A and ,O, and the number of multiply-adds to form .S. What does that cost look like as the sequence length n=m grows?
QK⊤ is ,(n×d)(d×m)=n×m, and so is .S.K⊤ is ,d×m, and dividing by the scalar d keeps the shape.
A is .n×m.The softmax is applied to each row separately and keeps each row's length.
O=AV is .(n×m)(m×dv)=n×dv.The inner dimensions agree: each query position gets a weighted sum of the m value rows.
Sij=d1∑lQilKjl takes d multiply-adds, and there are nm entries.Each score is the dot product of query row i with key row ,j, both of length .d.
;S,A:n×m;;O:n×dv;S costs nmd multiply-adds, quadratic in the sequence lengthWith n=m the cost is :n2d: doubling the sequence length quadruples it. O=AV costs another ,nmdv, and A itself has n2 entries to store for the backward pass, so time and memory both grow quadratically.
Problem 2
Compute .∇VL.
.Oic=∑jAijVjc.Index form of ,O=AV, which is linear in V for fixed .A.
Vjc appears in Oic for every ,i, with coefficient ,Aij, and in no entry of another column.Column c of O is built from column c of V only, and every query row uses key row .j.
.∂L/∂Vjc=∑iGicAij.L depends on V only through ;O; by step 2 the chain rule sums over the rows i of O in column .c.
,∑iAijGic=(A⊤G)jc, with shapes .(m×n)(n×dv)=m×dv.The sum runs over A's row index, which a product can only contract if A is transposed.
∇VL=A⊤G ()m×dv)The shape of .V. The pattern is general: for ,C=PR, the gradient with respect to the right factor is .P⊤∇CL.
Problem 3
Compute .G~=∇AL.
Aij appears in Oic for every ,c, with coefficient ,Vjc, and in no other row of .O.Problem 2, step 1: row i of A builds only row i of .O.
.∂L/∂Aij=∑cGicVjc.L depends on A only through ,O, and step 1 limits the sum to row .i.
,∑cGicVjc=(GV⊤)ij, with shapes .(n×dv)(dv×m)=n×m.The sum runs over V's column index, so V is transposed.
G~=∇AL=GV⊤ ()n×m)The shape of .A. The companion pattern to Problem 2: for ,C=PR, the gradient with respect to the left factor is .(∇CL)R⊤.
Problem 4
Compute ∇SL from G~ and ,A, using the softmax Jacobian row by row, and write it as one matrix expression with no m×m Jacobians.
Write ,si,ai and g~i∈Rm for row i of ,S,A and ,G~, as columns.
,ai=softmax(si), and ai depends on no other row of .S.The softmax is taken row by row, so row i of S reaches L only through row i of .A.
.∇siL=(diag(ai)−aiai⊤)g~i=ai⊙(g~i−(ai⊤g~i)1).The softmax page's row result with a=ai and upstream ,∇aiL=g~i, row i of ;G~; by step 1, ai is the only path from si to ,L, so no other row's upstream enters.
ai⊤g~i=∑jAijG~ij is entry i of ,rowsum(A⊙G~),.n×1.A dot product of two rows is the sum along that row of their elementwise product.
Row i of G~−rowsum(A⊙G~)1⊤ is ,(g~i−(ai⊤g~i)1)⊤, with shapes .(n×1)(1×m)=n×m.The outer product with 1⊤ copies entry i of the column across row ,i, so each row subtracts its own scalar.
∇SL=A⊙(G~−rowsum(A⊙G~)1⊤),n×m, the shape of :S: step 2 stacked, since an elementwise product acts row by row. Each row's Jacobian would be ,m×m, and the whole matrix's ;nm×nm; neither is built, and the cost is .O(nm). In array code the rowsum is a sum along the last axis with the dimension kept, so it broadcasts.
Problem 5
Compute .∇QL.
.Sij=d1∑lQilKjl.Index form of ,S=QK⊤/d, which is linear in Q for fixed .K.
Qil appears in Sij for every ,j, with coefficient ,Kjl/d, and in no other row of .S.Query row i is dotted with every key row, and builds only row i of .S.
.∂L/∂Qil=d1∑j(∇SL)ijKjl=d1((∇SL)K)il.L depends on Q only through ;S; step 2 limits the chain rule to row .i. The sum runs over K's row index, which is also the column index of ,∇SL, so no transpose is needed.
∇QL=d1(∇SL)K ()n×d),(n×m)(m×d), the shape of .Q. Problem 3's left-factor pattern with ,C=S,P=Q/d and R=K⊤ gives ;∇PL=(∇SL)K; since ,P=Q/d, each Qil enters L through Pil times ,d1, so .∇QL=d1∇PL.
Problem 6
Compute .∇KL.
Kjl appears in Sij for every ,i, with coefficient ,Qil/d, and in no other column of .S.Problem 5, step 1: key row j is dotted with every query row and builds only column j of .S.
.∂L/∂Kjl=d1∑i(∇SL)ijQil.L depends on K only through ,S, and step 1 limits the chain rule to column .j.
,∑i(∇SL)ijQil=((∇SL)⊤Q)jl, with shapes .(m×n)(n×d)=m×d.The sum runs over the row index of ,∇SL, so it is transposed.
∇KL=d1(∇SL)⊤Q ()m×d)The shape of .K. Equivalently, S⊤=KQ⊤/d has K as its left factor, and .∇S⊤L=(∇SL)⊤.
Problem 7
Self-attention: ,Q=XWQ,,K=XWK,.V=XWV. Compute ,∇WQL,,∇WKL,∇WVL and .∇XL.
Here ,m=n,WQ,WK∈Rdmodel×d and .WV∈Rdmodel×dv.,∇QL,∇KL and ∇VL are Problems 5, 6 and 2, each taken with the other two inputs held fixed.
,∇WQL=X⊤∇QL,.(dmodel×n)(n×d)=dmodel×d.WQ reaches L only through ,Q=XWQ, where it is the right factor: Problem 2's pattern.
∇WKL=X⊤∇KL and ,∇WVL=X⊤∇VL,dmodel×d and .dmodel×dv.The same argument for K=XWK and ;V=XWV; each weight matrix feeds one projection only.
The part of ∇XL through Q is ,∇QLWQ⊤,.(n×d)(d×dmodel)=n×dmodel.In ,Q=XWQ,X is the left factor: Problem 3's pattern.
The parts through K and V are ∇KLWK⊤ and ,∇VLWV⊤, both .n×dmodel.The same pattern; ∇KL is n×d because .m=n.
;∇WQL=X⊤∇QL;;∇WKL=X⊤∇KL;;∇WVL=X⊤∇VL;∇XL=∇QLWQ⊤+∇KLWK⊤+∇VLWV⊤X feeds all three projections, and the chain rule adds the contributions of every path from a variable to .L. Each gradient has the shape of its variable.
Problem 8
A causal mask sets Sij=−∞ for j>i before the softmax. What are the masked entries of ,A, and what are the masked entries of ?∇SL? Show it from Problem 4.
Here ,m=n, so S is square and the mask keeps the diagonal and everything below it.
Aij=eSij/∑j′≤ieSij′ for every .j.Row-wise softmax; the masked terms of the denominator are ,e−∞=0, so only j′≤i remain, and the j′=i term makes the sum positive.
Aij=0 for ,j>i, and the unmasked entries of each row sum to .1.The numerator is ;e−∞=0; the softmax renormalizes over the positions a query may see.
.(∇SL)ij=Aij(G~ij−∑j′Aij′G~ij′).Entry (i,j) of Problem 4's formula.
For j>i the bracket is finite and the factor Aij is ,0, so .(∇SL)ij=0.G~=GV⊤ is finite, so the bracket is too; zero times a finite number is exactly zero, not a small number.
The row sum ∑j′Aij′G~ij′ has zero terms at the masked .j′.Step 2: those Aij′ are .0.
masked ,Aij=0, and since ,∇SL=A⊙(⋯), masked (∇SL)ij=0 exactly; the unmasked entries follow Problem 4 with the masked columns contributing nothing to the row sumsSo no gradient reaches Q or K through a masked score: in Problems 5 and 6 those entries of ∇SL multiply by zero. In code the −∞ is often a large negative number such as ;−109; its exponential underflows to 0 in floating point, so the same holds.
Problem 9
If the entries of q and k are independent with mean 0 and variance ,1, what is ?Var(q⊤k)? What does dividing by d achieve, and where does the factor appear in the backward pass?
Here q,k∈Rd are one query row and one key row as columns, so q⊤k=∑lqlkl is a score before scaling.
.E[qlkl]=E[ql]E[kl]=0.ql and kl are independent, so the expectation of the product factors, and each mean is .0.
.Var(qlkl)=E[ql2kl2]−02=E[ql2]E[kl2]=1.Independence again; ,E[ql2]=Var(ql)+E[ql]2=1, and likewise for .kl.
.Var(∑lqlkl)=∑lVar(qlkl)=d.The terms for different l are functions of disjoint sets of independent entries, so they are independent and their variances add.
.Var(q⊤k/d)=d/d=1.Scaling a random variable by λ scales its variance by ,λ2, here .λ2=1/d.
Without the scaling the scores have standard deviation ,d,8 at ,d=64, so the softmax of a row is concentrated on a few keys, often one.The softmax weights of two scores differ by the factor ,exp(Δ), where Δ is the gap between them. At this spread the top two scores of a row are typically 2 to 3 units apart, and Δ=3 already gives .exp(3)≈20.
The more concentrated a row ,a, the smaller ,diag(a)−aa⊤, so ∇SL shrinks and less gradient reaches Q and .K.At a one-hot row, a standard basis vector ,u,,diag(u)−uu⊤=0, and the entries aj(1−aj) and −ajaj′ go to 0 as a approaches .u.
In the backward pass d appears only where S is differentiated with respect to Q and .K.,∇VL,G~ and ∇SL are computed from ,A,G and V (Problems 2 to 4), and none of their formulas contains .d.
;Var(q⊤k)=d; dividing by d makes the scores order 1 so the softmax does not saturate; the same d1 multiplies ∇QL and ∇KL (Problems 5–6) and nothing elseThe projection gradients of Problem 7 inherit it through ∇QL and ∇KL and add no second factor. Trained queries and keys are not independent unit-variance vectors, but the scaling keeps the scores at order 1 at initialization, where saturation would stop learning before it starts.
Problem 10
Multi-head: O=[O1⋯OH]WO with Oh∈Rn×dv and .WO∈RHdv×dmodel. Compute ∇WOL and .∇OhL.
In this problem O is the output of the multi-head layer, ,n×dmodel,G=∇OL has the same shape, and ,Oh, for ,h=1,…,H, is head h's output from the earlier problems. Write ,Ocat=[O1⋯OH],.n×Hdv.
,∇WOL=Ocat⊤G,.(Hdv×n)(n×dmodel)=Hdv×dmodel.O=OcatWO with WO the right factor: Problem 2's pattern.
,∇OcatL=GWO⊤,.(n×dmodel)(dmodel×Hdv)=n×Hdv.Ocat is the left factor: Problem 3's pattern.
Column (h−1)dv+c of Ocat is column c of ,Oh, for .c=1,…,dv.Concatenation places O1 in columns 1 to ,dv,O2 in columns dv+1 to ,2dv, and so on; each entry of Oh is exactly one entry of .Ocat.
;∇WOL=[O1⋯OH]⊤G;∇OhL = columns (h−1)dv+1 to hdv of GWO⊤,n×dv, the shape of .Oh. Concatenation copies entries without mixing them, so its backward pass slices the gradient into the same blocks. From there each head runs Problems 2 to 6 on its own block, and the heads' contributions to X add, as the three paths did in Problem 7.
Where this goes wrong
1. Gradient to V without the transpose
O=AV puts A on the left, and it is tempting to keep it there on the way back.
Oic=∑jAijVjc and G=∇OLRight so far: Problem 2, step 1.
“V is multiplied by ,A, so its gradient is A times the gradient coming back.”The shortcut that causes the mistake: the scalar rule, where the derivative of av is ,a, applied to matrices without asking which index the sum runs over.
∇VL=AGThe shapes (n×m)(n×dv) do not multiply unless ,n=m, and even then it is wrong. The index form, ,∂L/∂Vjc=∑iAijGic, puts the sum on A's row index, so the product is .A⊤G. In self-attention ,n=m, so the wrong version runs and trains on the wrong gradient.
2. Softmax Jacobian applied to the whole matrix
The softmax page's Jacobian is written for one vector, and flattening a matrix into a vector makes it look as though it applies at once. Write vec(B) for the column of length nm that stacks the rows of an n×m matrix B one after another (row-major).
∂a/∂s=diag(a)−aa⊤ for a=softmax(s)Right so far: the softmax page, Problem 3, for one row.
“Flatten S and A to vectors and use the same Jacobian.”The shortcut that causes the mistake: treating the row-wise softmax as one softmax over all nm scores.
∇SL=(diag(a)−aa⊤)g~ with a=vec(A) and g~=vec(G~)The softmax is per row: entries in different rows do not interact, so the true nm×nm Jacobian is block-diagonal, with the row Jacobians diag(ai)−aiai⊤ as blocks, and the row formula of Problem 4 is all there is. The flattened version subtracts ,a(a⊤g~), where a⊤g~ sums over every row, so each row is corrected by the whole matrix's total instead of its own.
3. Losing the 1/√d on the way back
The scaling is a fixed number with no parameter in it, and it is easy to decide that the backward pass can ignore it.
S=QK⊤/d and ∇SL from Problem 4Right so far.
“1/d is a constant, not something we train, so it needs no gradient.”The analogy that causes the mistake: treating the scaling as part of the data, when it is part of the function S of Q and .K. A constant needs no gradient of its own, but it still multiplies the gradient of everything it scales.
∇QL=(∇SL)KS contains the factor, so its derivative with respect to Q does too: d1(∇SL)K (Problem 5). The error makes ∇QL (and ,∇KL, if the factor is dropped there too) d times too large, 8 times at ,d=64, which under plain SGD acts like a larger learning rate for WQ (and )WK) and passes a check that only compares directions.
4. Only one path from X
Problem 7 has one input and three projections, and it is easy to follow only the one the problem starts with.
∇QLWQ⊤ is the gradient reaching X through QRight so far: Problem 7, step 3.
“X is the query input, so its gradient comes back through the queries.”The analogy that causes the mistake: cross-attention, where the queries come from X and the keys and values from another sequence, so the query path really is the only one.
∇XL=∇QLWQ⊤In self-attention X enters through ,Q,K and ,V, and the chain rule sums over every path: the terms ∇KLWK⊤ and ∇VLWV⊤ are missing. The shape is right, so nothing fails; the layers below simply train on part of their gradient.
5. Masking A instead of S
Problem 8 shows the masked weights are zero, and zeroing them directly looks like a shortcut to the same place. Write M for the n×n matrix with Mij=1 for j≤i and 0 otherwise.
The causal mask must make Aij=0 for j>iRight so far: Problem 8, step 2.
“Compute the full softmax, then zero the entries a query may not see.”The shortcut that causes the mistake: masking the output of the softmax instead of its input.
A←A⊙M after the softmaxRows no longer sum to ,1, so the forward pass is wrong before the backward pass starts. Worse, each denominator still contains eSij for the future positions ,j>i, so the kept weights depend on later keys and gradient flows into them: the model can see the future. Mask the scores with −∞ and the softmax renormalizes.
Print this set: attention-backward.pdf (problems, answers, and worked solutions on separate pages).