I used to treat linear algebra like a checklist: learn the definition, memorize the formula, move on. It worked well enough for passing quizzes, but when I started reading ML papers and implementing models, the notation still felt disconnected from anything real.
That changed after I watched 3Blue1Brown’s Essence of Linear Algebra.
What is linear algebra, really?
Linear algebra is the math of lists of numbers and machines that update those lists in a predictable way. A vector can be an arrow on a plane, but in machine learning it is more often a description of something: a house might be $x=[1200,3]$ for square feet and bedrooms; a token might be a list of 768 learned features. A matrix is the machine: feed it a vector, get another vector out. Grant Sanderson’s series made that picture stick for me — a matrix is not a grid of symbols, it is a transformation of space that stretches, rotates, or shears every input direction. Once I held that image, eigenvalues, determinants, and change of basis stopped feeling like magic notation and started feeling like consequences of the same idea.
flowchart LR
V["Input vector x"] --> M["Matrix A"]
M --> O["Output vector Ax"]
O --> D["Dot products measure alignment"]
O --> E["Eigenvectors = directions that only scale"]
The rest of this post walks one small $2\times2$ example from start to finish — vectors, matrix multiply, dot product, determinant, eigenvalue — with every arithmetic step written out. I will come back to where these ideas show up in models like the ones I am building in AL-1.
What is a vector?
A vector is an ordered list of numbers. In two dimensions I write
The first entry is how far I move along the horizontal axis; the second is how far along the vertical axis. I can read the same object as an arrow from the origin to the point $(4,5)$, or as a list of features. Both views are the same data; geometry gives me intuition, lists give me code.
How does a matrix transform a vector?
A matrix packages several output directions at once. I will use the same matrix throughout this post:
Row 1 says: “to build the first output coordinate, take $2$ times the first input plus $1$ times the second.” Row 2 says: “take $0$ times the first input plus $3$ times the second.” Applying $A$ to $x=[4,5]^T$ means computing each output entry separately.
First output coordinate:
Second output coordinate:
So the transformed vector is
Geometrically, the point $(4,5)$ lands at $(13,15)$. I can also read the same calculation in column language: the columns of $A$ are where the basis vectors $\hat{\imath}=[1,0]^T$ and $\hat{\jmath}=[0,1]^T$ get sent. The input $[4,5]^T$ says “take 4 copies of the first column plus 5 copies of the second”:
Neural-network layers usually think in the row view (weighted sums into each neuron); animations usually think in the column view (where basis arrows go). Same multiplication, two pictures.
One complete example: vectors through eigenvalues
I want one thread of numbers I can follow without switching examples. I keep $A=\begin{bmatrix}2&1\0&3\end{bmatrix}$ and $x=\begin{bmatrix}4\5\end{bmatrix}$ from above, then ask what else this same matrix tells me.
Matrix multiply (recap)
Dot product on two directions from the same setup
Take the first column of $A$ as vector $a$ and a separate direction $b$:
The dot product multiplies matching entries and adds:
Length of $a$:
Length of $b$:
The geometric formula $a\cdot b = |a||b|\cos\theta$ gives:
A large positive dot product means the vectors point roughly the same way; zero would mean perpendicular; negative would mean they face opposite directions. That is why dot products score similarity in search, recommendations, embeddings, and transformer attention.
Determinant: how much area scales
Before inverses and eigenvalues, I check whether $A$ squashes space. For a $2\times2$ matrix,
A unit square becomes a parallelogram with six times the area. If $\det(A)=0$, some direction collapses to a line or point and information is lost forever — no inverse can undo that.
Eigenvalues: which directions only get stretched
An eigenvector $v$ is a direction $A$ does not rotate — it only scales by some number $\lambda$:
I find $\lambda$ by solving $\det(A - \lambda I)=0$.
Set the determinant to zero:
So one eigenvalue is
and the other is
Eigenvector for $\lambda_1 = 2$. Solve $(A - 2I)v = 0$:
The first row gives
Pick $v_1 = 1$:
Check:
Eigenvector for $\lambda_2 = 3$. Solve $(A - 3I)v = 0$:
The first row gives
Pick $v_1 = 1$:
Check:
That is the full chain on one matrix: input vector, multiply, dot product for alignment, determinant for area scaling, eigenpairs for preserved directions. When I apply $A$ many times, an eigenvector direction grows or shrinks by $\lambda^k$ — which is why eigenvalues matter for stability, PCA, and worrying about exploding or vanishing signals in deep networks.
How do dot products measure alignment?
The numeric thread above already computed $a\cdot b=6$. The extra idea is projection: how much of one vector lies along another. For $a=[2,1]$ and $b=[3,4]$ (a different pair, same mechanics),
Projection of $b$ onto $a$:
Least-squares regression is the same picture in higher dimensions: when no exact solution exists, project the target onto the space of predictions the model can actually produce.
How does matrix multiplication compose transformations?
If matrix $B$ acts on $x$ first and matrix $A$ acts on the result, the combined effect is $ABx$ — rightmost matrix hits $x$ first. Order matters: rotate then stretch is not generally the same as stretch then rotate, so $AB \neq BA$ in most cases.
Shapes encode the contract. If $B$ is $3\times2$ and $x$ is $2\times1$, then
Matrix $A$ multiplying that result must have three columns. Dimension matching is not bureaucratic bookkeeping; it says the output space of one map must equal the input space of the next.
As a tiny composition with the same $A$, let
(swap the two coordinates). Entry $(AB)_{11}$:
Entry $(AB)_{12}$:
Entry $(AB)_{21}$:
Entry $(AB)_{22}$:
So
Applying to $x=[4,5]^T$ means swap first ($Bx=[5,4]^T$), then apply $A$ — one pipeline, one matrix product.
What do determinants and inverses tell you?
I already found $\det(A)=6$. When $\det(A)\neq 0$, an inverse exists that undoes the transformation. For this $A$,
Solving $Ax=b$ is $x = A^{-1}b$ in principle. In real code I use a numerical solver instead of forming the inverse explicitly — faster and more stable — but the picture is “run the transformation backward.”
What is a change of basis?
A basis is the coordinate language I use to describe vectors. The physical arrow does not move when I change basis; only the numbers describing it change. If the columns of $P$ are new basis vectors expressed in the old coordinates, then
and
Fourier transforms pick a frequency basis; PCA picks variance-aligned axes; token embeddings learn a coordinate system where semantic relationships become geometry later layers can exploit.
Where does machine learning use linear algebra?
Dense layers are $y = Wx + b$: each row of $W$ is a dot product against the input, exactly the row view of matrix multiply. Embedding tables store one vector per token ID. Attention scores query–key alignment with dot products and computes all pairs at once with matrix multiplication. Convolutions apply the same small linear map to every local patch. Regression and PCA lean on projection and eigenvectors of covariance matrices. Backpropagation threads Jacobians — local linear approximations of how small input changes affect outputs — through the graph; that is why I sanity-check gradients with numerical checking before trusting a new backward() implementation.
The AL-1 build path I am following assumes all of this is muscle memory: autograd → gradient checker → BPE → embeddings → positional encoding → attention. Linear algebra is the floor under every step.
Who should watch the 3Blue1Brown series?
If you are heading into machine learning, computer vision, or anything drowning in matrix notation, and symbols still feel like alphabet soup, the playlist is worth the time. You do not need a PhD — just patience and willingness to pause when an animation does not land on the first pass.
I still rewatch individual chapters when a course throws something abstract at me (SVD, change of basis, dual spaces). It is faster than rereading a textbook from zero. The animations gave me a place to put $Ax=b$ and weight matrices in PyTorch; the pencil work — one $2\times2$ example at a time — made the symbols trustworthy. If you are in the same boat I was in, watch the full Essence of Linear Algebra playlist, work through a tiny matrix by hand, and then go build something that actually multiplies matrices. That combination is what finally made it click for me.
Support the writing
If this post helped, a coffee keeps the deep dives, paper picks, and news digests coming.
Buy Me a Coffee