Practice sets for the maths behind machine learning

Original problems with full worked solutions, for adults getting their calculus and linear algebra back to do deep learning properly: matrix calculus, Jacobians and the chain rule, and the softmax Jacobian, with backprop by hand and the transformer pieces to follow. Each set shows every line of working, then the specific places the derivation goes wrong. Every set has a printable PDF.

• Matrix calculus

• Matrix calculus conventions

  Ten problems on layout, shapes and the gradients of Ax, xᵀAx, ‖Ax − b‖², traces and Frobenius norms, with full worked solutions and the mistakes that come from mixing conventions.

• Jacobians and the chain rule

  Ten problems on Jacobians of elementwise and affine maps, the order of the chain rule, vector-Jacobian products, and gradients through one and two layers, with worked solutions and the shape mistakes that hide inside them.

• Backprop by hand

• The softmax Jacobian

  Ten problems on softmax: its partial derivatives, the Jacobian diag(s) − ssᵀ and its rank, the s − y gradient of cross-entropy, log-softmax, soft labels, temperature and the batched version, with worked solutions and the mistakes that produce the wrong gradient.

Sets are added in curriculum order. Matrix calculus first, then backprop by hand, then the transformer pieces.