Practice / Linear algebra

Linear transformations and change of basis

Ten problems on linear maps and their matrices: a matrix built from the images of e₁ and e₂, a test for linearity, rotations and reflections composed in both orders, a projection with its kernel, image and rank–nullity count, coordinates in a new basis, a reflection across y = 2x built from diag(1, −1), the matrix of a map in a new basis two ways, what similar matrices share, the determinant as an area scale and an orientation, and PCA coordinates and whitening in an orthonormal basis, with worked solutions and the mistakes of writing the images as rows, composing in reading order, multiplying by P instead of P⁻¹, using Pᵀ for P⁻¹ when the basis is not orthonormal and treating x ↦ Ax + b as linear.

Before you start

A matrix is a linear map written down in a particular basis. Change the basis and the same map gets a different matrix, sometimes a much simpler one: a reflection across a slanted line becomes $\operatorname{diag}(1, -1)$ and a covariance matrix becomes diagonal in the coordinates PCA picks. These ten problems build the matrix of a map from what it does to the standard basis, test whether a map is linear at all, compose maps in the right order, take a projection apart into its kernel and image, and then change basis, first for a single vector and then for a whole map. They end with the determinant as an area scale and with PCA coordinates and whitening. The five mistakes at the end are the ones that still produce a matrix: the images written as rows, a composition multiplied in reading order, coordinates found with $P$ instead of $P^{-1}$ $P^\top$ used as $P^{-1}$ for a basis that is not orthonormal, and an affine map treated as linear.

Vectors are columns; $(3, -2)^\top$ is the column with entries $3$ and $-2$ written on one line to save space. $e_1 = (1, 0)^\top$ and $e_2 = (0, 1)^\top$ are the standard basis of $\mathbb{R}^2$
A map $T: \mathbb{R}^n \to \mathbb{R}^m$ is linear when $T(u + v) = T(u) + T(v)$ and $T(cu) = cT(u)$ for all vectors $u, v$ and numbers $c$ Then $T(x) = Ax$ for the $m\times n$ matrix $A$ whose column $j$ is $T(e_j)$
$AB$ means “apply $B$ then $A$ ”: $(AB)x = A(Bx)$
The kernel of $A$ is $\{x : Ax = 0\}$ the image is $\{Ax\}$ and for $A$ with $n$ columns, $\dim\ker A + \operatorname{rank}A = n$ (rank–nullity).
A basis $B = \{b_1, \ldots, b_n\}$ of $\mathbb{R}^n$ has basis matrix $P = [\,b_1 \;\cdots\; b_n\,]$ with the basis vectors as columns. The coordinates $[v]_B = c$ are the numbers with $v = c_1b_1 + \cdots + c_nb_n$ that is $Pc = v$ so $[v]_B = P^{-1}v$
The matrix of a linear map $T(x) = Ax$ in the basis $B$ is $[T]_B = P^{-1}AP$ its column $j$ is $[T(b_j)]_B$ Matrices $A$ and $P^{-1}AP$ are called similar.
A basis is orthonormal when its vectors have length $1$ and are pairwise orthogonal; then $P^\top P = I$ so $P^{-1} = P^\top$
For $2\times2$ matrices, $\begin{bmatrix}a&b\\c&d\end{bmatrix}^{-1} = \dfrac{1}{ad-bc}\begin{bmatrix}d&-b\\-c&a\end{bmatrix}$ when $ad - bc \neq 0$

Builds on: Vectors in 3-D: dot and cross products, lines and planes, Gaussian elimination, inverses and determinants by hand

Problems

·
A linear map $T: \mathbb{R}^2 \to \mathbb{R}^2$ sends $e_1$ to $(2, 1)^\top$ and $e_2$ to $(-1, 3)^\top$ Find its matrix $A$ and $T\big((3, -2)^\top\big)$
·
Which of these maps $\mathbb{R}^2 \to \mathbb{R}^2$ are linear? For each linear one, give its matrix.
$\begin{aligned} T_1(x, y) &= (x + 2y,\ 3x - y + 1)\\ T_2(x, y) &= (|x|,\ y)\\ T_3(x, y) &= (2x - y,\ x) \end{aligned}$
··
Let $R$ rotate the plane by $90^\circ$ counterclockwise and $F$ reflect it across the $x$ -axis. Find $R$ $F$ $RF$ and $FR$ and describe each product geometrically. Which one is “reflect first, then rotate”?
··
Let $\Pi$ be the orthogonal projection onto the line through $u = (1, 2)^\top$ so $\Pi x = \dfrac{u^\top x}{u^\top u}\,u$ Find the matrix of $\Pi$ its kernel and image, and check rank–nullity and $\Pi^2 = \Pi$
··
Let $B = \{b_1, b_2\}$ with $b_1 = (1, 1)^\top$ and $b_2 = (1, 2)^\top$ Find the coordinates $[v]_B$ of $v = (3, 5)^\top$ and check them.
···
Find the matrix, in the standard basis, of the reflection $T$ across the line $y = 2x$ Use a basis in which $T$ is $\operatorname{diag}(1, -1)$
···
Let $A = \begin{bmatrix}1&2\\0&3\end{bmatrix}$ and use the basis $B$ of Problem 5. Find $[T]_B$ for $T(x) = Ax$ in two ways: column by column from $[T(b_j)]_B$ and as $P^{-1}AP$
··
Show that similar matrices $A$ and $P^{-1}AP$ have the same characteristic polynomial, and hence the same eigenvalues, trace and determinant. Then show that $I$ and $N = \begin{bmatrix}1&1\\0&1\end{bmatrix}$ have the same trace and determinant but are not similar.
··
Let $A = \begin{bmatrix}1&3\\2&1\end{bmatrix}$ What is the area of the image of the triangle with vertices $(0, 0)$ $(1, 0)$ and $(0, 1)$ Does $A$ keep or reverse orientation? What is the area scale factor of $RA$ with $R$ the rotation of Problem 3?
···
A two-feature data set has covariance $\Sigma = \begin{bmatrix}5&2\\2&2\end{bmatrix}$ Its unit eigenvectors are $q_1 = (2, 1)^\top/\sqrt5$ and $q_2 = (1, -2)^\top/\sqrt5$ With $Q = [\,q_1\; q_2\,]$ find the PCA coordinates $c = [x]_Q$ of $x = (1, 3)^\top$ the covariance of the data in those coordinates, and the whitened coordinates $z = \Lambda^{-1/2}c$ where $\Lambda$ is the diagonal matrix of eigenvalues.

Answers

$A = \begin{bmatrix}2&-1\\1&3\end{bmatrix}$ $T\big((3, -2)^\top\big) = (8, -3)^\top$
Only $T_3$ is linear, with matrix $\begin{bmatrix}2&-1\\1&0\end{bmatrix}$ $T_1$ is affine, not linear, and $T_2$ is not linear
$RF = \begin{bmatrix}0&1\\1&0\end{bmatrix}$ reflection across $y = x$ is “reflect, then rotate”; $FR = \begin{bmatrix}0&-1\\-1&0\end{bmatrix}$ reflection across $y = -x$ is “rotate, then reflect”
$\Pi = \tfrac15\begin{bmatrix}1&2\\2&4\end{bmatrix}$ $\ker\Pi = \operatorname{span}\{(2, -1)^\top\}$ $\operatorname{im}\Pi = \operatorname{span}\{(1, 2)^\top\}$ $1 + 1 = 2$ $\Pi^2 = \Pi$
$[v]_B = (1, 2)^\top$ $v = b_1 + 2b_2$
$A = \tfrac15\begin{bmatrix}-3&4\\4&3\end{bmatrix}$
$[T]_B = \begin{bmatrix}3&4\\0&1\end{bmatrix}$ both ways
$\det(P^{-1}AP - \lambda I) = \det(A - \lambda I)$ so eigenvalues, trace and determinant are shared; $I$ and $N$ share them but are not similar, because $I$ is similar only to itself
Area $\tfrac52$ $A$ reverses orientation ( $\det A = -5$ $RA$ also scales areas by $5$ and reverses orientation
$c = (\sqrt5, -\sqrt5)^\top$ covariance $\operatorname{diag}(6, 1)$ uncorrelated; $z = (\sqrt{5/6}, -\sqrt5)^\top \approx (0.913, -2.236)^\top$ with covariance $I$

Worked solutions

Problem 1

A linear map $T: \mathbb{R}^2 \to \mathbb{R}^2$ sends $e_1$ to $(2, 1)^\top$ and $e_2$ to $(-1, 3)^\top$ Find its matrix $A$ and $T\big((3, -2)^\top\big)$

$(3, -2)^\top = 3e_1 - 2e_2$ Every vector is a combination of the standard basis, with its entries as the coefficients.
$T\big((3, -2)^\top\big) = 3T(e_1) - 2T(e_2)$ Linearity: $T$ of a combination is the same combination of the images.
$= 3(2, 1)^\top - 2(-1, 3)^\top = (8, -3)^\top$ $(6 + 2,\ 3 - 6)$
$A = \begin{bmatrix}2&-1\\1&3\end{bmatrix}$ Column $j$ is $T(e_j)$ so $Ae_j$ picks out exactly that image.
$A = \begin{bmatrix}2&-1\\1&3\end{bmatrix}$ $T\big((3, -2)^\top\big) = (8, -3)^\top$ Check: $A(3, -2)^\top$ has rows $2\cdot3 + (-1)(-2) = 8$ and $1\cdot3 + 3(-2) = -3$ Two images determine the whole map: that is what linearity buys.

Problem 2

Which of these maps $\mathbb{R}^2 \to \mathbb{R}^2$ are linear? For each linear one, give its matrix.

\begin{aligned} T_1(x, y) &= (x + 2y,\ 3x - y + 1)\\ T_2(x, y) &= (|x|,\ y)\\ T_3(x, y) &= (2x - y,\ x) \end{aligned}

$T_1(0, 0) = (0, 1)$ A linear map sends $0$ to $0$ because $T(0) = T(0\cdot0) = 0\cdot T(0) = 0$ $T_1$ fails, because of the constant $+1$
$T_2(-1, 0) = (1, 0)$ but $-T_2(1, 0) = (-1, 0)$ Scaling by $c = -1$ is not preserved, so $T_2$ is not linear, even though $T_2(0) = 0$
$T_3$ is linear: each output entry is a combination of $x$ and $y$ with no constant.Then both rules hold entry by entry.
$T_3(e_1) = (2, 1)^\top$ $T_3(e_2) = (-1, 0)^\top$ Read off with $(x, y) = (1, 0)$ and $(0, 1)$
Only $T_3$ is linear, with matrix $\begin{bmatrix}2&-1\\1&0\end{bmatrix}$ $T_1$ is affine, not linear, and $T_2$ is not linear $T_1(x) = Ax + b$ with $b = (0, 1)^\top$ A dense layer $Wx + b$ is affine in the same way; it becomes linear only with $b = 0$ or after appending a constant $1$ to $x$

Problem 3

Let $R$ rotate the plane by $90^\circ$ counterclockwise and $F$ reflect it across the $x$ -axis. Find $R$ $F$ $RF$ and $FR$ and describe each product geometrically. Which one is “reflect first, then rotate”?

$R = \begin{bmatrix}0&-1\\1&0\end{bmatrix}$ $R(e_1) = e_2 = (0, 1)^\top$ and $R(e_2) = -e_1 = (-1, 0)^\top$ are its columns.
$F = \begin{bmatrix}1&0\\0&-1\end{bmatrix}$ $F$ keeps $e_1$ and sends $e_2$ to $-e_2$
$RF = \begin{bmatrix}0&1\\1&0\end{bmatrix}$ Row times column; or by columns: $RF e_1 = R e_1 = e_2$ and $RF e_2 = R(-e_2) = e_1$
$RF$ swaps $x$ and $y$ it is the reflection across $y = x$ $RF(x, y)^\top = (y, x)^\top$
$FR = \begin{bmatrix}0&-1\\-1&0\end{bmatrix}$ so $FR(x, y)^\top = (-y, -x)^\top$ the reflection across $y = -x$ $FRe_1 = F e_2 = -e_2$ and $FRe_2 = F(-e_1) = -e_1$
$RF = \begin{bmatrix}0&1\\1&0\end{bmatrix}$ reflection across $y = x$ is “reflect, then rotate”; $FR = \begin{bmatrix}0&-1\\-1&0\end{bmatrix}$ reflection across $y = -x$ is “rotate, then reflect” $(RF)x = R(Fx)$ the factor next to $x$ acts first. Matrix products are composed right to left, and $RF \neq FR$ Both have determinant $-1$ as a rotation times a reflection must.

Problem 4

Let $\Pi$ be the orthogonal projection onto the line through $u = (1, 2)^\top$ so $\Pi x = \dfrac{u^\top x}{u^\top u}\,u$ Find the matrix of $\Pi$ its kernel and image, and check rank–nullity and $\Pi^2 = \Pi$

$\Pi x = \dfrac{u(u^\top x)}{u^\top u} = \dfrac{uu^\top}{u^\top u}\,x$ A scalar times a vector can be written with the scalar on the right, and $u(u^\top x) = (uu^\top)x$
$\Pi = \tfrac15\begin{bmatrix}1&2\\2&4\end{bmatrix}$ $u^\top u = 1 + 4 = 5$ and $uu^\top$ has entries $u_iu_j$
$\ker\Pi = \operatorname{span}\{(2, -1)^\top\}$ $\Pi x = 0$ exactly when $u^\top x = x_1 + 2x_2 = 0$ the vectors orthogonal to $u$
$\operatorname{im}\Pi = \operatorname{span}\{u\}$ so $\operatorname{rank}\Pi = 1$ Every output is a multiple of $u$ and $\Pi u = u$ so every multiple is reached.
$1 + 1 = 2$ the number of columns.Rank–nullity: $\dim\ker\Pi + \operatorname{rank}\Pi = n$
$\Pi^2 = \dfrac{u(u^\top u)u^\top}{(u^\top u)^2} = \dfrac{uu^\top}{u^\top u} = \Pi$ $u^\top u$ is a number and cancels once. Projecting a second time changes nothing.
$\Pi = \tfrac15\begin{bmatrix}1&2\\2&4\end{bmatrix}$ $\ker\Pi = \operatorname{span}\{(2, -1)^\top\}$ $\operatorname{im}\Pi = \operatorname{span}\{(1, 2)^\top\}$ $1 + 1 = 2$ $\Pi^2 = \Pi$ $\det\Pi = (4 - 4)/25 = 0$ a projection that loses a direction squashes area to zero, and cannot be undone.

Problem 5

Let $B = \{b_1, b_2\}$ with $b_1 = (1, 1)^\top$ and $b_2 = (1, 2)^\top$ Find the coordinates $[v]_B$ of $v = (3, 5)^\top$ and check them.

$P = \begin{bmatrix}1&1\\1&2\end{bmatrix}$ The basis vectors are the columns.
$\det P = 2 - 1 = 1$ so $P^{-1} = \begin{bmatrix}2&-1\\-1&1\end{bmatrix}$ The $2\times2$ inverse, with nothing to divide by. $\det P \neq 0$ so $b_1, b_2$ really are a basis.
$[v]_B = P^{-1}v = (2\cdot3 - 5,\ -3 + 5)^\top = (1, 2)^\top$ $[v]_B$ solves $Pc = v$
$1\cdot b_1 + 2\cdot b_2 = (1 + 2,\ 1 + 4)^\top = (3, 5)^\top$ The check multiplies by $P$ the reverse direction.
$[v]_B = (1, 2)^\top$ $v = b_1 + 2b_2$ $P$ turns $B$ -coordinates into standard ones; $P^{-1}$ goes the other way. In the standard basis the coordinates of $v$ are its entries, $(3, 5)$ in $B$ the same vector is $(1, 2)$

Problem 6

Find the matrix, in the standard basis, of the reflection $T$ across the line $y = 2x$ Use a basis in which $T$ is $\operatorname{diag}(1, -1)$

Take $b_1 = (1, 2)^\top$ on the line and $b_2 = (-2, 1)^\top$ perpendicular to it. $b_1^\top b_2 = -2 + 2 = 0$ The reflection keeps the line and flips the perpendicular direction.
$T(b_1) = b_1$ and $T(b_2) = -b_2$ so $[T]_B = \operatorname{diag}(1, -1) = D$ Column $j$ of $[T]_B$ is $[T(b_j)]_B$ here $(1, 0)^\top$ and $(0, -1)^\top$
$A = PDP^{-1}$ with $P = \begin{bmatrix}1&-2\\2&1\end{bmatrix}$ $[T]_B = P^{-1}AP$ solved for $A$ multiply by $P$ on the left and $P^{-1}$ on the right.
$\det P = 1 + 4 = 5$ so $P^{-1} = \tfrac15\begin{bmatrix}1&2\\-2&1\end{bmatrix}$ The $2\times2$ inverse. The columns of $P$ are orthogonal but have length $\sqrt5$ so $P^{-1} = \tfrac15P^\top$ not $P^\top$
$PD = \begin{bmatrix}1&2\\2&-1\end{bmatrix}$ $D$ on the right scales column $2$ of $P$ by $-1$
$PDP^{-1} = \tfrac15\begin{bmatrix}1 - 4&2 + 2\\2 + 2&4 - 1\end{bmatrix}$ Row times column with $\begin{bmatrix}1&2\\-2&1\end{bmatrix}$ then the $\tfrac15$
$A = \tfrac15\begin{bmatrix}-3&4\\4&3\end{bmatrix}$ Checks: $A(1, 2)^\top = \tfrac15(5, 10)^\top = (1, 2)^\top$ $A(-2, 1)^\top = (2, -1)^\top$ $A^2 = I$ and $\det A = -1$ It matches the formula for the reflection across a line at angle $\varphi$ with $\cos2\varphi = -\tfrac35$ and $\sin2\varphi = \tfrac45$ when $\tan\varphi = 2$

Problem 7

Let $A = \begin{bmatrix}1&2\\0&3\end{bmatrix}$ and use the basis $B$ of Problem 5. Find $[T]_B$ for $T(x) = Ax$ in two ways: column by column from $[T(b_j)]_B$ and as $P^{-1}AP$

$T(b_1) = A(1, 1)^\top = (3, 3)^\top = 3b_1 + 0b_2$ So column $1$ of $[T]_B$ is $(3, 0)^\top$ $b_1$ happens to be an eigenvector of $A$
$T(b_2) = A(1, 2)^\top = (5, 6)^\top$ Row times column: $1 + 4$ and $0 + 6$
$[T(b_2)]_B = P^{-1}(5, 6)^\top = (10 - 6,\ -5 + 6)^\top = (4, 1)^\top$ $P^{-1}$ from Problem 5. Check: $4b_1 + b_2 = (5, 6)^\top$
$AP = \begin{bmatrix}3&5\\3&6\end{bmatrix}$ Its columns are $T(b_1)$ and $T(b_2)$ from steps 1 and 2.
$P^{-1}(AP) = \begin{bmatrix}6 - 3&10 - 6\\-3 + 3&-5 + 6\end{bmatrix}$ Row times column with $P^{-1} = \begin{bmatrix}2&-1\\-1&1\end{bmatrix}$ this applies $P^{-1}$ to each column of $AP$ which is step 3 done for both columns at once.
$[T]_B = \begin{bmatrix}3&4\\0&1\end{bmatrix}$ both waysBoth $A$ and $[T]_B$ have trace $4$ and determinant $3$ so the same eigenvalues $1$ and $3$ they are one map written in two bases. Read right to left, $P^{-1}AP$ says: turn $B$ -coordinates into standard ones ( $P$ apply the map ( $A$ turn the result back into $B$ -coordinates ( $P^{-1}$

Problem 8

Show that similar matrices $A$ and $P^{-1}AP$ have the same characteristic polynomial, and hence the same eigenvalues, trace and determinant. Then show that $I$ and $N = \begin{bmatrix}1&1\\0&1\end{bmatrix}$ have the same trace and determinant but are not similar.

$P^{-1}AP - \lambda I = P^{-1}(A - \lambda I)P$ $\lambda I = \lambda P^{-1}P = P^{-1}(\lambda I)P$ then factor.
$\det\big(P^{-1}(A - \lambda I)P\big) = \det P^{-1}\,\det(A - \lambda I)\,\det P = \det(A - \lambda I)$ $\det$ of a product is the product of the $\det$ s, and $\det P^{-1} = 1/\det P$
The eigenvalues are the roots of the characteristic polynomial, and for a $2\times2$ matrix it is $\lambda^2 - (\operatorname{tr}A)\lambda + \det A$ So equal polynomials mean equal eigenvalues, trace and determinant; for $n\times n$ the trace and determinant are still coefficients of the polynomial.
$\operatorname{tr}I = \operatorname{tr}N = 2$ and $\det I = \det N = 1$ Both are triangular with diagonal $(1, 1)$
$P^{-1}IP = P^{-1}P = I$ for every invertible $P$ So the only matrix similar to $I$ is $I$ itself, and $N \neq I$
$\det(P^{-1}AP - \lambda I) = \det(A - \lambda I)$ so eigenvalues, trace and determinant are shared; $I$ and $N$ share them but are not similar, because $I$ is similar only to itselfEqual invariants are necessary for similarity, not sufficient. $N$ is the shear of the eigenvalues page: it has one eigenvector direction where $I$ has two.

Problem 9

Let $A = \begin{bmatrix}1&3\\2&1\end{bmatrix}$ What is the area of the image of the triangle with vertices $(0, 0)$ $(1, 0)$ and $(0, 1)$ Does $A$ keep or reverse orientation? What is the area scale factor of $RA$ with $R$ the rotation of Problem 3?

$\det A = 1 - 6 = -5$ $ad - bc$
A linear map scales every area by $|\det A|$ It sends the unit square to the parallelogram spanned by its columns, whose area is $|\det A|$ and every region is a limit of small squares.
The image triangle has area $5\cdot\tfrac12 = \tfrac52$ The original triangle has area $\tfrac12$ Its image has vertices $(0, 0)$ $(1, 2)$ and $(3, 1)$ and $\tfrac12|1\cdot1 - 3\cdot2| = \tfrac52$ directly.
$\det A < 0$ so $A$ reverses orientation.Going $(0, 0) \to (1, 0) \to (0, 1)$ is counterclockwise; the image path $(0, 0) \to (1, 2) \to (3, 1)$ turns clockwise.
$\det(RA) = \det R\,\det A = 1\cdot(-5) = -5$ A rotation has determinant $1$ it keeps both area and orientation.
Area $\tfrac52$ $A$ reverses orientation ( $\det A = -5$ $RA$ also scales areas by $5$ and reverses orientationThe sign of the determinant records orientation and its size records the area scale. In a normalizing flow, $\log|\det|$ of each layer's Jacobian is exactly this area (volume) scale, and it enters the log-likelihood.

Problem 10

A two-feature data set has covariance $\Sigma = \begin{bmatrix}5&2\\2&2\end{bmatrix}$ Its unit eigenvectors are $q_1 = (2, 1)^\top/\sqrt5$ and $q_2 = (1, -2)^\top/\sqrt5$ With $Q = [\,q_1\; q_2\,]$ find the PCA coordinates $c = [x]_Q$ of $x = (1, 3)^\top$ the covariance of the data in those coordinates, and the whitened coordinates $z = \Lambda^{-1/2}c$ where $\Lambda$ is the diagonal matrix of eigenvalues.

$\Sigma q_1 = (12, 6)^\top/\sqrt5 = 6q_1$ and $\Sigma q_2 = (1, -2)^\top/\sqrt5 = q_2$ so $\Lambda = \operatorname{diag}(6, 1)$ Check the eigenvectors before using them: $(10 + 2,\ 4 + 2)$ and $(5 - 4,\ 2 - 4)$
$q_1^\top q_2 = (2 - 2)/5 = 0$ and $q_1^\top q_1 = q_2^\top q_2 = 1$ so $Q^{-1} = Q^\top$ The basis is orthonormal, so no inverse needs computing.
$c = Q^\top x = \big((2 + 3)/\sqrt5,\ (1 - 6)/\sqrt5\big)^\top = (\sqrt5, -\sqrt5)^\top$ Each coordinate is a dot product $q_j^\top x$ Check: $\|c\|^2 = 10 = \|x\|^2$ since an orthonormal change of basis keeps lengths.
In $Q$ -coordinates the data are $c = Q^\top x$ with covariance $Q^\top\Sigma Q$ For a linear map $M$ $\operatorname{Cov}(Mx) = M\Sigma M^\top$ here $M = Q^\top$
$Q^\top\Sigma Q = Q^{-1}\Sigma Q = \Lambda = \operatorname{diag}(6, 1)$ This is $[T]_Q$ for $T(x) = \Sigma x$ in the eigenvector basis the matrix is diagonal.
$z = \Lambda^{-1/2}c = (\sqrt5/\sqrt6,\ -\sqrt5)^\top \approx (0.913, -2.236)^\top$ Divide coordinate $j$ by $\sqrt{\lambda_j}$ its covariance is $\Lambda^{-1/2}\Lambda\Lambda^{-1/2} = I$
$c = (\sqrt5, -\sqrt5)^\top$ covariance $\operatorname{diag}(6, 1)$ uncorrelated; $z = (\sqrt{5/6}, -\sqrt5)^\top \approx (0.913, -2.236)^\top$ with covariance $I$ PCA is a change to an orthonormal basis that diagonalizes $\Sigma$ whitening then rescales each new axis to variance $1$ The rescaling is not orthogonal, so $\|z\| \neq \|x\|$

Where this goes wrong

1. Images written as rows

The images $T(e_1)$ and $T(e_2)$ are usually listed one after the other, and it is natural to write them into the matrix the same way, line by line.

$T(e_1) = (2, 1)^\top$ $T(e_2) = (-1, 3)^\top$ Right so far: Problem 1.
“Write the first image in the first row and the second in the second.”The habit that causes the mistake: filling a matrix row by row, the way it is read.
$A = \begin{bmatrix}2&1\\-1&3\end{bmatrix}$ so $T\big((3, -2)^\top\big) = (4, -9)^\top$ That is $A^\top$ $Ae_1$ is the first column of $A$ so the images must be columns; the right answer is $(8, -3)^\top$ The check is cheap: multiply the matrix by $e_1$ and see whether $T(e_1)$ comes out.

2. Composing in reading order

“Reflect, then rotate” names the reflection first, and the product is easy to write in the same order.

$R = \begin{bmatrix}0&-1\\1&0\end{bmatrix}$ and $F = \begin{bmatrix}1&0\\0&-1\end{bmatrix}$ Right so far: Problem 3.
“Reflect, then rotate: $F$ then $R$ so $FR$ ”The habit that causes the mistake: writing the factors left to right in the order they happen.
“Reflect, then rotate” $= FR$ which sends $e_1$ to $(0, -1)^\top$ $(FR)x = F(Rx)$ rotates first. Reflecting $e_1$ across the $x$ -axis leaves it alone, and rotating gives $(0, 1)^\top$ so the right product is $RF$ (Problem 3). The factor nearest $x$ acts first: in a network $W_2W_1x$ the layer $W_1$ is applied first.

3. Coordinates found with P instead of P⁻¹

$P$ is the matrix that comes with the basis, and multiplying by it is the obvious thing to do with a new basis.

$P = \begin{bmatrix}1&1\\1&2\end{bmatrix}$ and $v = (3, 5)^\top$ Right so far: Problem 5.
“To express $v$ in the new basis, apply the basis matrix to it.”The shortcut that causes the mistake: $P$ goes from $B$ -coordinates to standard ones, not the other way.
$[v]_B = Pv = (8, 13)^\top$ Then $8b_1 + 13b_2 = (21, 34)^\top$ not $v$ The coordinates solve $Pc = v$ so $[v]_B = P^{-1}v = (1, 2)^\top$ (Problem 5). Rebuilding $v$ from the claimed coordinates catches the error at once.

4. Pᵀ used as P⁻¹ for a basis that is not orthonormal

Orthogonal basis vectors look like the case where the inverse is the transpose.

$P = \begin{bmatrix}1&-2\\2&1\end{bmatrix}$ with orthogonal columns, and $v = (3, 1)^\top$ Right so far: the basis of Problem 6.
“The columns are orthogonal, so $P^{-1} = P^\top$ ”The rule that causes the mistake: $P^\top P = I$ needs columns of length $1$ as well, and these have length $\sqrt5$
$[v]_B = P^\top v = (5, -5)^\top$ $P^\top P = 5I$ so $P^{-1} = \tfrac15P^\top$ and $[v]_B = (1, -1)^\top$ indeed $b_1 - b_2 = (3, 1)^\top$ The transpose gives coordinates $5$ times too large, and $5b_1 - 5b_2 = (15, 5)^\top$ Normalize the basis first, as in Problem 10, or divide by the squared lengths.

5. An affine map treated as linear

The recipe “column $j$ is $T(e_j)$ ” works so reliably that it gets applied before checking that the map is linear.

$T_1(x, y) = (x + 2y,\ 3x - y + 1)$ so $T_1(e_1) = (1, 4)^\top$ and $T_1(e_2) = (2, 0)^\top$ Right so far: the map of Problem 2, evaluated correctly.
“Put the images in the columns.”The habit that causes the mistake: the recipe assumes linearity, and $T_1(0) = (0, 1)^\top \neq 0$
$T_1 = \begin{bmatrix}1&2\\4&0\end{bmatrix}$ so $T_1(1, 1) = (3, 4)$ The true value is $T_1(1, 1) = (3, 3)$ Each column carries the constant $b$ so the matrix applied to $(1, 1)^\top$ adds it twice, where $T_1$ adds it once. The map is $Ax + b$ with $A = \begin{bmatrix}1&2\\3&-1\end{bmatrix}$ and $b = (0, 1)^\top$ read $A$ from $T(e_j) - T(0)$

Print this set: linear-transformations-and-change-of-basis.pdf (problems, answers, and worked solutions on separate pages).