EulerPad
Practice sets for the math behind machine learning
Matrix calculus
- Matrix calculus conventions
Ten problems on layout, shapes and the gradients of Ax, xᵀAx, ‖Ax − b‖², traces and Frobenius norms, with full worked solutions and the mistakes that come from mixing conventions.
- Jacobians and the chain rule
Ten problems on Jacobians of elementwise and affine maps, the order of the chain rule, vector-Jacobian products, and gradients through one and two layers, with worked solutions and the shape mistakes that hide inside them.
Regression
- Regression gradients: linear, logistic and softmax
Ten problems on the gradients of linear, logistic and softmax regression in matrix form: the 2/N of the mean squared error, the normal equations and ridge, the logistic gradient Xᵀ(σ(Xw) − y)/N, its Hessian and convexity, and the softmax-regression gradient Xᵀ(S − Y)/N with shapes, with worked solutions and the mistakes that lose a factor or a transpose.
- Logistic regression: log-odds, decision boundaries and thresholds
Ten problems on what a logistic regression model says, rather than how it is trained: probabilities from a logit, weights read as odds ratios, the decision boundary as a line with its normal and a point's distance from it, a threshold other than one half, the scale of the weights, separable data where the weights grow without bound and a ridge penalty that stops them, the linear boundary of two Gaussian classes with unequal priors, XOR and the feature that fixes it, a circular boundary from squared features, and the boundaries between three softmax classes, with worked solutions and the mistakes of multiplying a probability by an odds ratio, leaving out the norm in a distance, putting the threshold on the logit, trusting a small gradient on separable data and placing the boundary halfway between the classes when the priors differ.
Backprop by hand
- The softmax Jacobian
Ten problems on softmax: its partial derivatives, the Jacobian diag(s) − ssᵀ and its rank, the s − y gradient of cross-entropy, log-softmax, soft labels, temperature and the batched version, with worked solutions and the mistakes that produce the wrong gradient.
- Activation functions and their derivatives
Ten problems on the activations a network is built from: sigmoid, tanh, ReLU and leaky ReLU, softplus, SiLU and GELU, their derivatives, the same derivatives written in terms of the stored output, the backward pass through an elementwise layer and its gradient bounds, the behavior near zero, and the numerically stable cross-entropy from logits, with worked solutions and the mistakes that drop a product rule or differentiate the wrong variable.
- One-hidden-layer backprop, the whole backward pass
Ten problems assembling the full backward pass of a one-hidden-layer classifier: the two deltas, every parameter gradient with its shape, the gradient to the input, the batched version with the 1/N, a ReLU variant and an L2 penalty, with worked solutions and the mistakes that swap shapes or drop factors.
Transformer pieces
- Layer norm backward
Ten problems on the backward pass of layer normalization: the gradients of the mean and variance, the centering projection, the Jacobian of x̂, the input gradient and why it is orthogonal to 1 and, up to ε, to x̂, the batched version and RMSNorm, with worked solutions and the mistakes that treat μ and σ as constants.
- Attention backward
Ten problems on the backward pass of scaled dot-product attention: the gradients to V, A, the scores, Q and K, the projections in self-attention, the causal mask, why the scores are divided by √d, and the multi-head output projection, with worked solutions and the mistakes that transpose the wrong matrix.
Minibatches
- Batched backprop: dense layers on a minibatch
Ten problems on backprop with a minibatch: the dense layer XW + 1bᵀ and its gradients XᵀdY, dYᵀ1 and dY Wᵀ, why a sum over the batch becomes a matrix product, the 1/N of a mean loss, activations and masked softmax cross-entropy on a batch, micro-batch accumulation and sequence batches, with worked solutions and the mistakes that transpose, double-count or drop a factor.
- Batch norm backward
Ten problems on the backward pass of batch normalization: the gradients of γ, β and the batch statistics, the compact input gradient and why each feature's column of it sums to zero, batch norm as layer norm on the transpose, inference with running statistics, a batch of one and the role of ε, with worked solutions and the mistakes that hold the batch statistics constant or take them over the wrong axis.
Convolutions and recurrences
- Convolution backward: CNN backprop
Ten problems on the backward pass of a convolutional layer: the kernel gradient as a correlation of the input with dy, the input gradient as a full convolution with the flipped kernel, the im2col and Toeplitz matrix views, 2-D and multi-channel layers on a batch, stride, same padding and pooling, with worked solutions and the mistakes that forget the flip or the sum over positions.
- Backprop through time: RNN and LSTM gradients
Ten problems on backpropagation through time: the one-step Jacobians of a vanilla RNN, the backward recursion for the hidden-state gradients, the weight gradients as sums over time, a two-step example by hand, vanishing and exploding gradients as a product of Jacobians, many-to-one classification, and the LSTM cell's backward pass and cell-state path, with worked solutions and the mistakes that drop a time step or a factor.
Probability for ML
- Counting: permutations, combinations and the binomial theorem
Ten counting problems: the size of a hyperparameter grid and the number of ordered augmentation pipelines by the multiplication principle, choosing three features of ten against ranking them, splitting twelve examples into three minibatches with a multinomial coefficient, benchmark suites with at least one text task by the complement, the coefficient of x² in (2x − 3)⁵ from the binomial theorem, counting feature subsets and even-sized subsets with (1 + 1)ⁿ and (1 − 1)ⁿ, sharing ten GPU nodes among four jobs by stars and bars, the chance of a hash collision among thirty keys and the birthday bound, derangements of five shuffled labels by inclusion–exclusion, and drawing a test batch with exactly two positives by Vandermonde's identity, with worked solutions and the mistakes of counting arrangements when order does not matter, dropping the power on the coefficient inside a binomial term, choosing one item and then the rest, putting one bar too many in stars and bars, and adding overlapping cases instead of using inclusion–exclusion.
- Discrete random variables: expectation, variance and the binomial, geometric and Poisson
Ten problems on discrete random variables: expectation and variance from a probability table, a fair die and Var(aX + b), the binomial probabilities and its mean and variance from indicators, linearity of expectation without independence in the matching problem, the geometric distribution and memorylessness, Poisson counts and the Poisson limit of the binomial, the laws of total expectation and variance, and the expected number of distinct examples in a bootstrap sample, with worked solutions and the mistakes of squaring the mean, dropping the binomial coefficient and adding probabilities that overlap.
- Continuous random variables: densities, CDFs, expectation and the uniform, exponential and normal
Ten problems on continuous random variables: normalizing a density and integrating it for a probability, a CDF turned into a density, a median and a mean with an infinite variance, expectation and variance by integration, the uniform distribution and a density above 1, the exponential's CDF, mean, median and memorylessness, standardizing a normal with z-values and a percentile, the density of Y = X² through its CDF, inverse-CDF sampling and the minimum of independent exponentials, with worked solutions and the mistakes of reading a density as a probability, dropping the |dx/dy| in a change of variables, taking the exponential's mean as λ, dividing by σ² and taking the mean as the median.
- The normal distribution and z-scores
Ten problems on the normal distribution: standardizing and reading Φ with its symmetry, the 68-95-99.7 rule for an initialization scale, a classifier threshold from an inverse z-score and the true positive rate it gives, z-scores that compare results on different scales, linear transforms, sums and differences of independent normals, the density's peak and inflection points, the normal approximation to a binomial with a continuity correction, a two-component Gaussian mixture, and the false alarms of a |z| > 3 outlier rule, with worked solutions and the mistakes of using a two-sided cutoff for a one-sided rate, adding standard deviations, dropping the continuity correction, reading a density as a probability and treating a mixture as one normal.
- Bayes' rule and conditional probability
Ten problems on conditional probability and Bayes' rule: the product rule and the law of total probability, a diagnostic test at a 1% base rate, posterior odds as prior odds times a likelihood ratio, two tests in sequence, naive Bayes by hand, a counterexample separating independence from conditional independence, the Beta–Bernoulli posterior, precision as a posterior, and a calibrated score corrected for label shift, with worked solutions and the mistakes that swap a conditional or skip the normalizer.
- Maximum likelihood estimation: derivations by hand
Ten problems on maximum likelihood estimation: the Bernoulli, categorical, Poisson and Gaussian MLEs by hand, the curvature of the log-likelihood, why the variance MLE divides by N and is biased, least squares and cross-entropy as negative log-likelihoods, MAP with a Gaussian or Laplace prior as ridge and lasso with the exact λ, and the uniform MLE that no derivative finds, with worked solutions and the mistakes that maximize the wrong thing.
- Bayesian inference: conjugate priors, posterior means and MAP estimates
Ten problems on Bayesian inference for a continuous parameter: a Beta posterior from a uniform prior and its MAP estimate, the posterior mean as a weighted average of the prior mean and the data, how the posterior narrows, updating one batch at a time, a Gamma prior on a Poisson error rate, the mean of a normal with known variance and its 95% credible interval, MAP regression with a Gaussian prior as ridge regression, the chance of the next two successes, a Beta prior built from a mean and a standard deviation, and add-one smoothing as a Dirichlet posterior mean, with worked solutions and the mistakes of squaring the posterior mean, forgetting the N in the data's precision, swapping the Gamma updates, inverting the ridge penalty and counting the first batch twice.
- Entropy, cross-entropy and KL divergence
Ten problems on entropy, cross-entropy and the Kullback–Leibler divergence: the Bernoulli entropy and its maximum, why the uniform distribution maximizes entropy, bits, nats and perplexity, cross-entropy as entropy plus KL, label smoothing, a proof that KL is non-negative, forward and reverse KL, the KL between two Gaussians and the VAE term with its gradients, with worked solutions and the mistakes that swap the arguments or drop a factor.
- Variance, covariance and correlation
Ten problems on variance, covariance and correlation: the shortcut formulas, the variance of a sum, bilinearity and the quadratic form aᵀΣa, the bounds on a correlation, a dependent pair with zero covariance, the covariance matrix as a positive semidefinite matrix, the covariance of a linear map and whitening, the variance of an average of correlated variables, the sample covariance as XᵀHX, and the correlation matrix with the bound on a common correlation, with worked solutions and the mistakes that add variances of correlated terms or read zero covariance as independence.
- The law of large numbers, Markov's inequality and Chebyshev's inequality
Ten problems on the law of large numbers and the inequalities behind it: Markov's bound on a latency, Chebyshev's bound against the normal's exact tail, Chebyshev proved from Markov with the distribution that makes it tight, the bound against a uniform's exact tail, the weak law for die rolls, the sample size for estimating a coin's bias, a Monte Carlo estimate of π, the batch size for a minibatch gradient, the variance of an average of correlated samples and why proportions settle while counts spread, with worked solutions and the mistakes of using Markov on a variable that can be negative, putting σ where σ² belongs, dividing by n instead of √n, averaging correlated samples as if they were independent and expecting a coin to even out.
- Sampling distributions and the central limit theorem: standard errors, confidence intervals and the √N rule
Ten problems on the sample mean and the central limit theorem: the mean and standard error of an average and of a sum, the sum of 100 dice with a continuity correction, a poll's margin, the sample size for a target error, a 95% confidence interval, the error of a Monte Carlo estimate, minibatch gradient noise, a sum of exponential waits, an A/B test, and a rare event where the normal approximation fails, with worked solutions and the mistakes of dividing by N instead of √N, standardizing a mean with σ, doubling the data to halve the error, adding standard errors, and trusting the normal when Np is small.
- Hypothesis tests and p-values: z and t tests, power and multiple comparisons
Ten problems on hypothesis tests: a one-sample z test on a latency benchmark with its two-sided p-value, one-sided against two-sided tests and why the side is chosen first, the confidence interval as the set of nulls a test keeps, a t test on nine measurements with the t critical value, an A/B test of two conversion rates with the pooled standard error, the power of a test against a given effect, the sample size for 80% power, McNemar's test for two classifiers scored on the same examples, the false-positive rate of twenty tests and the Bonferroni fix, and an exact permutation test on three runs per model, with worked solutions and the mistakes of reading the p-value as the probability that the null is true, choosing the side after seeing the data, using z instead of t on a small sample, taking one group's standard error for a difference, and testing twenty metrics and reporting the one that passed.
- Chi-square tests: goodness of fit, independence and the chi-square distribution
Ten problems on chi-square tests: the chi-square distribution as a sum of squared standard normals with its mean, variance and the 5% cutoff for two degrees of freedom, a coin whose chi-square statistic is the square of its z score, a goodness-of-fit test of a die rolled 120 times, a classifier's predicted label distribution against a target mix, a Poisson fit to hourly error counts with an estimated rate and a merged tail cell, a 2 × 2 A/B table whose chi-square equals the two-proportion z squared, a 3 × 2 table of model versions and user ratings, its Pearson residuals and Cramér's V, how the statistic grows with the sample size when the proportions stay fixed, and the G-test as twice N times a KL divergence, with worked solutions and the mistakes of running the test on percentages instead of counts, forgetting to subtract a degree of freedom for an estimated parameter, using rc − 1 degrees of freedom for a contingency table, dividing by the observed counts instead of the expected ones and reading a large p-value as proof that the model fits.
- Simple linear regression: the least-squares line, residuals and R²
Ten problems on fitting a straight line by least squares: the slope and intercept from sums of squares, residuals that sum to zero, the split of the total sum of squares and R² as the squared correlation, the slope from a correlation and two standard deviations, the standard error of the slope with a t test and confidence interval, what changes with the units, one point of high leverage, the normal equations from calculus, intervals for a mean response and for a new observation, and a negative R² on test data, with worked solutions and the mistakes of inverting the slope to predict x, dividing by n instead of n − 2, reading r as the fraction of variance explained, extrapolating far past the data and trusting training R² as a test score.
- The multivariate Gaussian: gradients and identities
Ten problems on the multivariate Gaussian: the log-density and its normalizing constant, the score −Σ⁻¹(x − μ), the maximum-likelihood mean and covariance through the gradients of log det Σ and tr(Σ⁻¹A), affine maps, the reparameterization trick, conditionals by the Schur complement and the product of two Gaussian densities, with worked solutions and the mistakes that put Σ where Σ⁻¹ or its square root belongs.
- Markov chains: transition matrices, stationary distributions and absorption
Ten problems on Markov chains: two-step probabilities from P², the distribution after n steps as a row vector times Pⁿ, the stationary distribution from πP = π, the two-state chain in closed form, how many steps until it has mixed from the second eigenvalue, expected return times 1/πᵢ and a hitting time, gambler's ruin and a checkout funnel with the fundamental matrix (I − Q)⁻¹, a random walk on a graph with detailed balance and a chain that is not reversible, and PageRank with damping, with worked solutions and the mistakes of multiplying P by a column vector, reading the stationary distribution off a right eigenvector, expecting a periodic chain to converge, using 1/πⱼ as a hitting time and multiplying win probabilities in gambler's ruin.
- Conditional expectation and the law of total variance
Ten problems on conditional expectation: E[Y | X = x] from a joint table, E[Y | X] as a random variable with its own mean and variance, the law of total variance on the same table, the mean and variance of a two-component Gaussian mixture, a random sum of purchases with a Poisson count, E[Y | X] from the joint density x + y, E[Y | X] as the best mean-squared-error predictor with a residual uncorrelated with every function of X, the conditional mean and variance of a bivariate Gaussian, Rao–Blackwell variance reduction for a Monte Carlo estimate, and the noise that dropout adds split from the spread of the data, with worked solutions and the mistakes of averaging the conditional variances only, evaluating E[Y | X] at the mean of X, inverting a regression line, taking E[N]σ² as the variance of a random sum and averaging conditional means without their weights.
Gradients of expectations
- The ELBO and the reparameterization trick
Ten problems on the evidence lower bound: its derivation from Jensen's inequality, the gap as a KL divergence to the posterior, the reconstruction-minus-KL form, the KL between diagonal Gaussians, the reparameterized gradients in μ, σ and log σ², the variance of the score-function estimator next to the reparameterized one, a linear Gaussian model solved in closed form with its tight bound, the one-sample gradient of a VAE step, and the Monte Carlo KL estimate and its variance, with worked solutions and the mistakes that reverse a KL, detach the sample or apply Jensen the wrong way.
- Policy gradients and the log-derivative trick
Ten problems on differentiating an expectation through its distribution: the log-derivative trick, the zero-mean score and why baselines are free, the softmax policy gradient in closed form and a gradient step with numbers, the variance of the one-sample estimator and the optimal baseline, the Gaussian policy's scores, trajectories with the dynamics dropping out, reward-to-go, the entropy bonus and the importance-sampled surrogate, with worked solutions and the mistakes that move the gradient inside the expectation, differentiate with respect to the action, or weight actions by past rewards.
Initialization and optimizers
- Xavier and He initialization: the variance by hand
Ten problems on weight initialization: the variance of a product and of a dense layer's output, why a ReLU halves the second moment, He's 2/n_in and Xavier's 2/(n_in + n_out) derived forwards and backwards, the signal scale through depth, fan-in for convolutions, the leaky-ReLU gain and uniform bounds, and residual stacks, with worked solutions and the mistakes that lose a factor of 2, 3 or 9.
- Momentum, RMSProp and Adam: optimizer updates by hand
Ten problems on optimizer update rules: gradient descent on a quadratic and the 2/λ_max learning-rate limit, momentum unrolled as a weighted sum of past gradients and its 1/(1 − β) step, the EMA form, Nesterov, AdaGrad and RMSProp, Adam's bias correction and its first step, two Adam steps by hand, and L2 penalty versus AdamW's decoupled weight decay, with worked solutions and the mistakes that change the step size without anyone noticing.
Calculus
- Evaluating limits: factoring, conjugates and limits at infinity
Ten limits evaluated exactly: 0/0 forms cleared by factoring, by a difference of cubes and by the conjugate, the difference quotient behind every derivative, sin 3x / x, one-sided limits that disagree, and limits at ±∞ including √(x²) = |x| and ∞ − ∞, with worked solutions and the mistakes that read 0/0 as an answer.
- Differentiation rules: chain, product, quotient
Ten single-variable derivatives that exercise the product, quotient and chain rules on the functions machine learning uses — exponentials, logs, sigmoid, tanh, softplus, nested trig, xˣ and an implicit curve — with worked solutions and the mistakes that drop the inner derivative or flip the quotient rule.
- Implicit differentiation: differentiate both sides, then solve for dy/dx
Ten implicit derivatives: the slope and tangent line of a circle and an ellipse, a curve with an xy term, an exponential and a log, a trig relation, the folium of Descartes, the horizontal and vertical tangents of a tilted ellipse, a second derivative, the derivative of arcsin, and logarithmic differentiation of a product and quotient, with worked solutions and the mistakes of dropping dy/dx, skipping the product rule, differentiating a constant, losing a y' term and treating y as a constant.
- Derivatives of inverse functions and inverse trig functions
Ten problems on the derivatives of inverse functions: the slope of an inverse at a point without a formula for the inverse, the derivatives of arccos and arctan from implicit differentiation, the chain rule through arctan and arcsin, an antiderivative of arcsin checked by differentiating, the derivative of the logit as the inverse of the sigmoid, the second derivative of an inverse, a branch that sets the sign, a camera tracking a rocket and the best place to stand in front of a painting, with worked solutions and the mistakes of reading f⁻¹ as 1/f, evaluating f′ at the wrong point, giving arccos a positive slope, dropping the inner derivative and choosing the wrong branch.
- L'Hôpital's rule and indeterminate forms
Ten limits by L'Hôpital's rule: 0/0 and ∞/∞, a rule applied twice, x ln x and the 0 log 0 = 0 of entropy, xˣ and (1 + 2/x)ˣ by taking logs, a difference of fractions, and a limit the rule cannot find, with worked solutions and the mistakes of using it on a limit that is not indeterminate and of reading 1^∞ as 1.
- Related rates: differentiate the relation, then substitute
Ten related-rates problems solved by differentiating an equation in time: a growing circle and sphere, a sliding ladder, a filling cone and trough, two cars, a shadow, a rising balloon's angle, the length of a moving vector, and the distance between clock hands, with worked solutions and the mistakes of substituting too early and dropping the chain rule.
- The mean value theorem and Rolle's theorem
Ten problems on the mean value theorem: Rolle's theorem on a cubic, the point c for x³ and for √x, two functions where a hypothesis fails, bounds on √101 and on sin 1.2 − sin 1.1, an equation with exactly one root, two cameras that prove a car was speeding, why equal derivatives give functions that differ by a constant, and the Lipschitz bounds of the logistic loss and its gradient, with worked solutions and the mistakes of applying the theorem to |x|, keeping a c outside the interval, calling a function constant on a domain with a gap, using a lower bound as an upper bound and saying where the car was speeding.
- Maxima and minima: critical points, the closed-interval method and optimization problems
Ten single-variable extremum problems: a cubic classified by the second-derivative test, a flat point that is not an extremum, a minimum the second-derivative test cannot see, the closed-interval method with an endpoint maximum, a cusp where the derivative is undefined, x e^(−x), an open box, the closest point on a curve, a can of least surface area, and the mean as the minimizer of squared error, with worked solutions and the mistakes of forgetting endpoints and cusps and of reading f′ = 0 as a guaranteed extremum.
- Riemann sums: left, right, midpoint and trapezoid, and the limit that defines the integral
Ten problems on Riemann sums: left and right sums for x² on [0, 2], the midpoint sum and its error, the trapezoid rule and Simpson's combination, over- and underestimates for 1/x, the limit of right sums for 2x + 1 and for x², a sum recognized as π/4, distance from a table of speeds, how many subintervals an error bound needs, and the area under an ROC curve with uneven steps, with worked solutions and the mistakes of using every point in a left sum, writing x_i = i/n on the wrong interval and assuming equal widths.
- The fundamental theorem of calculus: derivatives of integrals and definite integrals
Ten problems on both parts of the fundamental theorem: an accumulation function checked against its closed form, a chain-rule upper limit, a variable lower limit, two moving limits and the maximum they give, critical points of an integral, definite integrals with an absolute value, the derivative of a Gaussian and an exponential CDF, and a limit of an integral by L'Hôpital's rule, with worked solutions and the mistakes of a missing chain-rule factor, a lost minus sign and confusing F with f.
- Integration: substitution and parts
Ten integrals by substitution and by parts — including parts applied twice, a definite integral, an improper one, and the sigmoid whose antiderivative is softplus — with worked solutions and the mistakes that forget the du or pick u and dv the wrong way round.
- Area between curves
Ten areas between curves: top minus bottom between the intersection points, regions that cross in the middle, a region integrated in y, a log and an integration by parts, a horizontal line that halves an area, and the area between two probability densities, with worked solutions and the mistakes that give a negative area or a total of zero.
- Volumes of revolution: disks, washers and shells
Ten volumes by slicing: disks under y = √x, the cone and sphere formulas derived as checks, a washer between two curves, disks in y about the y-axis, shells about the y-axis, one solid by shells and by washers about x = 2, a washer about y = −1, a region that is one shell integral but two washer integrals, and a solid of square cross-sections, with worked solutions and the mistakes of squaring a difference of radii, forgetting a shifted axis and dropping the π.
- Arc length and surface area of revolution
Ten problems on the length of a curve and the area of a surface of revolution: a line checked against the distance formula, y = x^(3/2), the catenary, a curve built so that 1 + (y′)² is a perfect square, arc length in y, a circular arc that gives back the angle, y = x² by trig substitution, the sphere's 4πr², √x turned about the x-axis and x² turned about the y-axis, with worked solutions and the mistakes of taking the square root term by term, leaving out the slant factor and measuring the radius from the wrong axis.
- Parametric and polar curves: slopes, arc length and area
Ten problems on curves given by a parameter or in polar coordinates: the slope and tangent lines of x = t², y = t³ − 3t, its second derivative, an ellipse and its area, an arc length with a perfect square under the root, the cycloid's length and area, a path that runs along a segment and back, the circle r = 2 cos θ, the cardioid's area and length, the region between two polar curves, and the slope of a polar curve, with worked solutions and the mistakes of stopping at d/dt of dy/dx, dropping the absolute value in a speed and integrating over a curve traced twice.
- Conic sections: parabolas, ellipses and hyperbolas
Ten problems on conic sections: the focus and directrix of a parabola, the foci and eccentricity of an ellipse and the constant sum of distances, a hyperbola's foci, asymptotes and constant difference, completing the square to find the center, building equations from foci and asymptotes, classifying a conic with an xy term by its discriminant, rotating the axes to remove the xy term, a tangent line to an ellipse by implicit differentiation, a conic in polar form with its focus at the origin, and the area of an ellipse by integration, with worked solutions and the mistakes of using c² = a² + b² for an ellipse, taking the asymptote slope upside down, completing the square without the coefficient, reading a and b without the square roots and classifying by the signs of x² and y² alone.
- Partial fractions and integrating rational functions
Ten rational functions split into partial fractions and integrated: distinct linear factors by the cover-up rule, three factors, a repeated factor, an irreducible quadratic, an improper fraction that needs long division first, and the definite integral of 1/(x² − 1), with worked solutions and the mistakes that choose the wrong form or skip the division.
- Trigonometric substitution: sin, tan and sec, the triangle and the limits
Ten integrals by trigonometric substitution: an arcsine with new limits, the area formula for √(1 − x²), x³/√(4 − x²) two ways, 1/(1 + x²)² by x = tan θ, ln(1 + √2) from ∫ sec θ, a secant substitution read back off the triangle, completing the square first, the area of an ellipse, √(x² − 4)/x on [2, 4], and the normalizing constant of a Student-t distribution, with worked solutions and the mistakes of dropping the dx, keeping the x-limits and integrating tan²θ by the power rule.
- Improper integrals: infinite limits, infinite integrands and comparison
Ten improper integrals: 1/x³ to infinity, the p-test at infinity and at zero, an integrand that blows up inside the interval, ln x on (0, 1], 1/(1 + x²) over the whole line, x e^(−x²), a comparison that proves convergence, a partial-fraction limit that gives ln 2, and the Cauchy distribution whose mean does not exist, with worked solutions and the mistakes of integrating straight through a singularity, canceling ∞ − ∞ and comparing in the wrong direction.
- Separable differential equations: growth, decay, cooling and the logistic curve
Ten separable differential equations: exponential growth with a negative start, y′ = x/y, a solution that blows up in finite time, the equation whose solution is softplus, half-life, Newton's law of cooling, the sigmoid as the solution of y′ = y(1 − y), gradient flow on a quadratic loss, a constant solution lost by dividing, and a draining tank, with worked solutions and the mistakes of adding the constant too late, splitting a square root and forgetting where a solution stops.
- Linear first-order differential equations: the integrating factor
Ten linear first-order differential equations solved with an integrating factor: a constant forcing term, e^(2x) on the right, xy′ + 2y = x² after rewriting in standard form, a forcing term that needs integration by parts, sec x as the factor, one that is also separable, a salt tank with constant volume, a sinusoidal input and its lagging steady state, a tank that fills while it drains, and the Bernoulli substitution that turns the logistic equation into a linear one, with worked solutions and the mistakes of multiplying only one side, adding the constant after dividing and holding a changing volume constant.
- Second-order linear differential equations with constant coefficients
Ten second-order linear differential equations with constant coefficients: distinct real roots with an initial value problem, a repeated root and the extra factor of t, complex roots and a decaying oscillation, a spring's angular frequency, period and amplitude, underdamped, critically damped and overdamped springs, undetermined coefficients for a polynomial and an exponential input, the resonant case that needs t times the guess, a sinusoidal input with its steady-state amplitude and lag, superposition of two inputs, and the first-order system x′ = Ax whose eigenvalues are the characteristic roots, with worked solutions and the mistakes of giving a repeated root two copies of one exponential, flipping the sign of the decay, guessing a solution of the homogeneous equation and applying the initial conditions before the particular solution is added.
- Laplace transforms: the table, the shift rules and initial-value problems
Ten problems on Laplace transforms: e^(3t) and t from the definition with the region of convergence, a sum of a constant, a power and a sine by linearity and the table, the first shift theorem for e^(−2t) cos 3t and t³e^(4t), an inverse transform by partial fractions, an inverse by completing the square into a damped oscillation, a first-order initial-value problem, a second-order initial-value problem with nonzero initial conditions, the transform of a ramp switched on at t = 2, a spring switched on by a unit step at t = π, and the impulse response, step response and final value of y″ + 2y′ + 5y from the transfer function and the convolution theorem, with worked solutions and the mistakes of dropping the initial conditions from the derivative rule, transforming a product as the product of the transforms, shifting the wrong way, transforming u(t − c)f(t) as e^(−cs)F(s) and completing the square in the denominator only.
- Fourier series: coefficients, even and odd extensions, convergence and Parseval
Ten problems on Fourier series: the orthogonality integrals of sines and cosines on [−π, π], the square wave with its Gibbs overshoot, the sawtooth f(x) = x and its value at the jump, the cosine series of |x| and the sum of 1/(2k − 1)², the series of x² and the Basel sum π²/6, a function of period 4 with the coefficients over L, the half-range sine series of x(π − x) and a sum of alternating cubes, Parseval's identity for the sum of 1/n⁴, the complex form of e^x and the alternating sum of 1/(1 + n²), and the periodic response of y″ + 4y to a square wave, solved term by term, with worked solutions and the mistakes of writing a₀ instead of a₀/2, dividing by π when the period is 2L, taking the series to equal f at a jump, treating cos nπ as 0 and leaving the a₀ term out of Parseval's identity.
- Arithmetic and geometric sequences and sums
Ten problems on arithmetic and geometric sequences and their sums: a sequence from two of its terms, the sum of the multiples of 7 between 100 and 500, the first sum past 1000, a geometric sequence with a negative ratio, the derivation of the geometric sum formula, a repeating decimal as a fraction, sums in sigma notation with shifted indices, the weights of an exponential moving average and its bias correction, the sum of k xᵏ and the mean of a geometric distribution, and the monthly payment on a loan, with worked solutions and the mistakes of counting terms as upper minus lower, summing a geometric series with |r| ≥ 1, putting the wrong digits over 999, bias-correcting with 1 − β instead of 1 − βᵗ, and rounding the number of terms down.
- Series convergence tests: geometric, divergence, integral, comparison and alternating
Ten problems on infinite series: a geometric series that starts at n = 1, a telescoping sum, the divergence test and what it cannot say about 1/√n, the integral test on 1/(n ln² n) with bounds on the sum, a direct comparison for (2 + cos n)/n², a limit comparison, the alternating harmonic series and conditional convergence, how many terms an alternating error bound needs, the ratio test on n!/nⁿ, and the step-size conditions behind stochastic gradient descent, with worked solutions and the mistakes of starting a geometric sum at the wrong term, reading terms that go to 0 as convergence and comparing with a bigger divergent series.
- Taylor series and linear approximation
Ten problems on tangent lines and Taylor series — the sign of a linear approximation's error, the standard Maclaurin series, series by substitution, a Taylor polynomial about 4, a Lagrange remainder bound, a radius of convergence, and the softplus, log-sigmoid and quadratic-loss expansions that machine learning uses — with worked solutions and the mistakes that drop the n! or use a series outside its radius.
- Newton's method for roots and minima
Ten problems on Newton's method: the update read off the tangent line, square roots and cube roots by iteration, a reciprocal computed without dividing, the exact error identity that makes the digits double, the slow convergence at a double root and its fix, a starting point that cycles, an iteration that runs away, Newton's method for a minimum, a system of two equations with its Jacobian, and a normal quantile found by inverting the distribution function, with worked solutions and the mistakes of adding the step instead of subtracting it, finding a root of the function when its minimum is wanted, expecting quadratic convergence at a double root, seeding the reciprocal iteration too high and transposing the Jacobian.
- Partial derivatives, gradients and Hessians
Ten problems on functions of several variables: partial derivatives, the gradient as the direction of steepest ascent and the normal to a level set, directional derivatives, Hessians and the classification of critical points, the quadratic ½xᵀAx − bᵀx with a Newton step, log-sum-exp, second-order Taylor, the logistic-loss Hessian and convexity, with worked solutions and the mistakes that misread a Hessian.
- Lagrange multipliers
Ten problems on optimizing under constraints with Lagrange multipliers: why ∇f = λ∇g, extrema on a line and on a circle, an open box of least area, the closest point on a plane, xᵀAx on the unit sphere, the maximum-entropy and Gibbs distributions, the minimum-norm solution of Ax = b, λ as the sensitivity of the optimum and KKT with active and inactive constraints, with worked solutions and the mistakes that cancel a variable or drop a candidate.
- Double integrals: iterated integrals, switching the order and polar coordinates
Ten double integrals worked by hand: iterated integrals over a rectangle in both orders, a product that splits into two single integrals, a triangle and the region between y = x² and y = x, switching the order to integrate e^(y²) and sin(y)/y, polar coordinates over a disk and a quarter disk, the Gaussian integral √π, and a joint density with a probability, two expectations and a covariance, with worked solutions and the mistakes of swapping limits without redrawing the region, dropping the r in r dr dθ and squaring an integral with one variable.
- Triple integrals: iterated limits, cylindrical and spherical coordinates
Ten triple integrals worked by hand: a product over a box, the volume and centroid of a tetrahedron, limits read off a slanted plane, the solid under a paraboloid and its centroid in cylindrical coordinates, a moment of inertia, the volume of a ball in spherical coordinates, an ice-cream cone, the integral of ρ² over a ball, a change of order that makes an impossible integral easy, and the probability that a 3-D Gaussian lands in the unit ball, with worked solutions and the mistakes of dropping the r, leaving out sin φ, running φ to 2π, putting numbers where the inner limits need variables, and taking the cone's radius from the sphere.
- Line integrals and Green's theorem: work, potentials and circulation
Ten problems on line integrals and Green's theorem: the mass of a semicircular wire with ds = |r′(t)| dt, the work of a force along a parabola, one field integrated along two paths with different answers, the cross-partials test and a potential function, the fundamental theorem for line integrals on an awkward curve, the circulation around the unit circle by Green's theorem, the area of an ellipse from ½∮(x dy − y dx), outward flux and the divergence, the field around a hole that circulates 2π, and gradient descent against the rotating field of a two-player game, with worked solutions and the mistakes of dropping |r′(t)|, a clockwise boundary, the curl written backward, a zero curl read as a conservative field and dx = dt with dy left alone.
- Surface integrals and the divergence theorem: dS, flux and Stokes
Ten problems on surface integrals: the area of a sphere from |r_φ × r_θ|, the area of a paraboloid as a graph, the mass of a hemispherical shell, the upward flux through a paraboloid cap with the normal (−g_x, −g_y, 1), the outward flux through a sphere, the divergence theorem on a box and on a ball, the flux through a hemisphere found by closing it with a disk, Stokes' theorem on a hemisphere and its boundary circle, and Gauss's law for the field r/|r|³ with the origin inside and outside, with worked solutions and the mistakes of dropping |r_u × r_v|, the normal (g_x, g_y, 1), forgetting the disk that closes the surface, a boundary traced the wrong way for Stokes and the divergence theorem across a singularity.
Linear algebra
- Vectors in 3-D: dot and cross products, lines and planes
Ten problems on vectors in three dimensions: lengths, the dot product and the angle between two vectors, work and the component along a direction, the cross product and the area of a parallelogram, a triangle's area and the plane through three points, the volume of a parallelepiped from a determinant, a line meeting a plane, the distance from a point to a plane, the angle and line of intersection of two planes, the distance between skew lines and a linear classifier's signed distances, with worked solutions and the mistakes of the middle sign of the cross product, dividing by squared lengths, forgetting to divide by |n|, a plane through the wrong point and the product of lengths as a volume.
- Gaussian elimination, inverses and determinants by hand
Ten problems on solving linear systems by row reduction: a 2×2 and a 3×3 system on the augmented matrix, a zero pivot that needs a row swap, a free variable and an inconsistent row, the 2×2 inverse formula, a 3×3 inverse by Gauss–Jordan on [A | I], a determinant by row reduction, the rules for det(cA), det(AB) and det(A⁻¹), an LU factorization with forward and back substitution, and a singular matrix whose null space is a redundant feature, with worked solutions and the mistakes of updating only the left side, dropping 1/det and forgetting the sign of a swap.
- Linear transformations and change of basis
Ten problems on linear maps and their matrices: a matrix built from the images of e₁ and e₂, a test for linearity, rotations and reflections composed in both orders, a projection with its kernel, image and rank–nullity count, coordinates in a new basis, a reflection across y = 2x built from diag(1, −1), the matrix of a map in a new basis two ways, what similar matrices share, the determinant as an area scale and an orientation, and PCA coordinates and whitening in an orthonormal basis, with worked solutions and the mistakes of writing the images as rows, composing in reading order, multiplying by P instead of P⁻¹, using Pᵀ for P⁻¹ when the basis is not orthonormal and treating x ↦ Ax + b as linear.
- Eigenvalues and eigenvectors by hand
Ten problems on eigenvalues and eigenvectors worked by hand: 2×2 and 3×3 matrices, the characteristic polynomial, a rotation with complex eigenvalues, a shear with too few eigenvectors, diagonalization and powers, orthogonal eigenvectors of symmetric matrices, power iteration and singular values, with worked solutions and the classic sign and null-space mistakes.
- Quadratic forms and positive definite matrices
Ten problems on quadratic forms and positive definite matrices: writing a form as xᵀAx with a symmetric matrix, classifying definite, semidefinite and indefinite matrices by their eigenvalues, the leading principal minors of a 3×3 matrix, completing the square as LDLᵀ, a Cholesky factor by hand, the range of a parameter that keeps a matrix positive definite, the largest and smallest values on the unit circle, the axes of an ellipse, why AᵀA is positive semidefinite and when it is definite, and the Hessian of a ridge regression loss, with worked solutions and the mistakes of the unhalved cross term, a positive diagonal taken as proof, leading minors for a semidefinite test, the long axis along the largest eigenvalue and the extremes read off the diagonal.
- Matrix exponentials and linear systems of ODEs
Ten problems on matrix exponentials and linear systems of differential equations: solving x′ = Ax from the eigenvectors and the initial condition, e^(At) = Ve^(Λt)V⁻¹ for a 2×2 matrix, a repeated eigenvalue with one eigenvector and e^(At) = e^(λt)(I + Nt), the rotation e^(Jt) from the series and a decaying spiral, classifying four equilibria from the trace and determinant, a second-order equation rewritten as a first-order system, det e^(At) = e^(t tr A), two matrices with e^(A+B) ≠ e^A e^B, a two-state machine with rate matrix Q and P(t) = e^(Qt), and gradient flow on a quadratic loss compared with gradient descent, with worked solutions and the mistakes of exponentiating entry by entry, splitting e^(A+B) when A and B do not commute, writing V⁻¹e^(Λt)V, treating a defective matrix as diagonalizable and reading stability off the diagonal.
- Orthogonal bases, Gram–Schmidt and the QR decomposition
Ten problems on orthogonality worked by hand: dot products, lengths and angles, coordinates in an orthonormal basis by dot products, a Householder reflection and the properties of an orthogonal matrix, Gram–Schmidt in the plane and in three dimensions, the QR decomposition of a 3×2 and a 3×3 matrix with det A from R, least squares by solving Rx = Qᵀb, the projection QQᵀ and the orthogonal complement, a dependent column that Gram–Schmidt turns into zero, and why nearly collinear features break the normal equations but not QR, with worked solutions and the mistakes of skipping the division by a·a, projecting onto a column instead of q, transposing R and taking QQᵀ to be I.
- Least squares, projections and the SVD
Ten problems on projections, least squares and the singular value decomposition: projecting onto a line and a column space, fitting a line through three points, why the residual is orthogonal, an SVD computed by hand, reading rank and null space off it, the pseudoinverse and the best rank-one approximation, with worked solutions and the mistakes that confuse U with V or drop (AᵀA)⁻¹.
- PCA and the covariance matrix
Ten problems on principal component analysis worked on four points in the plane: centering and the covariance matrix, its eigen-decomposition, the variance along a direction and why the top eigenvector maximizes it, total and explained variance, scores and reconstructions, the reconstruction error as the discarded eigenvalues, why no other subspace does better, PCA from the SVD, decorrelated and whitened scores, and what feature scaling does to the components, with worked solutions and the mistakes that skip the centering, read directions from the wrong singular vectors or take eigenvalues in the wrong order.
Trigonometry
- Graphs of sine and cosine: amplitude, period and phase shift
Ten problems on the graphs of sine and cosine: amplitude, period, midline and range read off a formula, the phase shift of sin(2x − π/3) with B factored out, the five key points of one period, cosine as a shifted sine, an equation built from a maximum and the next minimum, a daylight model and the days above 14 hours, hertz against radians per second for a mains voltage, the period of sin 2x + cos 3x, why the order of a stretch and a shift matters, and the wavelengths of a transformer's positional encoding with the rotation that moves it k positions, with worked solutions and the mistakes of reading the phase shift before factoring out B, taking the period as 2πB, taking the maximum as the amplitude, dropping the midline's sign and evaluating in degrees.
- Trig identities
Ten trigonometric identities proved from the Pythagorean identity and the sum formulas: double and triple angles, power reduction, half-angle tangent, sum-to-product, tan of a sum, and writing a sin x + b cos x as a single sine, with worked solutions and the mistakes of working both sides and linearizing sin(x + y).
- Solving trig equations and the unit circle
Ten trigonometric equations solved exactly on [0, 2π) and in general: sin, cos and tan set to unit-circle values, quadratics in sin x, equations mixing 2x and x, a triple angle, and sin x + cos x = 1, with worked solutions and the mistakes that lose solutions by dividing or gain them by squaring.
- Law of sines and law of cosines
Ten triangles solved with the law of sines and the law of cosines: AAS, SAS and SSS, an obtuse angle found exactly, the ambiguous SSA case with two, one and no triangles, the area formula, a distance across a lake, and the angle between two vectors from their lengths, with worked solutions and the mistakes that lose the obtuse angle or the second triangle.
- Complex numbers in polar form: Euler's formula, De Moivre and roots of unity
Ten problems on complex numbers in polar form: the modulus and argument of −1 + √3i and −2 − 2i, polar back to rectangular, a product and a quotient that give cos(5π/12) exactly, (1 + i)^10 and (1 − √3i)^6 by De Moivre, the three cube roots of 8i, the fifth roots of unity and cos(2π/5), the triple-angle formulas for cos 3θ and sin 3θ, the integral of e^x cos x from the real part of e^((1+i)x), a rotation about a point other than the origin, and the rotary position embeddings that make an attention score depend only on the distance between positions, with worked solutions and the mistakes of taking arctan(b/a) in the wrong quadrant, adding the moduli of a product, keeping only one of the n roots, raising the real and imaginary parts to a power separately and taking the modulus of a sum as the sum of the moduli.
Sets are listed in curriculum order: matrix calculus, backprop by hand, the transformer pieces, minibatches and batch norm, convolutions and recurrent networks, probability for ML, gradients of expectations, initialization and optimizers first, then the calculus, linear algebra and trigonometry underneath them.