Reading sequence
AI from Scratch
The whole path in order — linear algebra intuition, tokenizers, embeddings, attention, the full transformer forward and backward pass, optimizers, and two complete builds (autograd in NumPy, GPT-2 in PyTorch). Each part assumes the one before it.
15 parts
244 min total
- Linear Algebra for Machine Learning: The Series That Made It Click A complete intuitive introduction to vectors, matrices, dot products, determinants, eigenvectors, basis changes, and how linear algebra powers machine learning.
- Backpropagation from Scratch: Build an Autograd Engine in Python How does backpropagation work? Build a working autograd engine from scratch in ~80 lines of Python — computation graphs, the chain rule, and reverse-mode autodiff — the same core idea behind PyTorch's .backward().
- Numerical Gradient Checking: How to Debug Your Autograd Engine Before Training GPT What is numerical gradient checking? Learn the central difference formula, why you never train with finite differences, and how to build a grad checker that catches bugs in your autograd before LayerNorm or attention.
- Byte Pair Encoding (BPE) Explained: How GPT Tokenizers Turn Text Into Numbers What is byte pair encoding (BPE)? Learn how GPT tokenizers split text into IDs — pretokenization, merge rules, byte-level vocab, and a from-scratch implementation you can actually debug.
- Token Embeddings Explained: How LLMs Turn Token IDs Into Meaning Vectors What is a token embedding? How LLMs map token IDs to learned vectors — the embedding matrix, gather lookup vs one-hot matmul, parameter counts, and a from-scratch NumPy implementation.
- Positional Encoding Explained: How Transformers Learn Word Order What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal position embeddings, why we add instead of concatenate, and a from-scratch NumPy implementation.
- Cross-Entropy Loss Explained: From Logits to Language Model Training What is cross-entropy loss? Derive softmax + negative log-likelihood step by step, work a 3-class numeric example, see why MSE fails for classification, and build the loss GPT training actually uses.
- Adam and AdamW Explained: How Modern LLMs Actually Update Their Weights How does the Adam optimizer work? Derive momentum, RMSprop, and bias correction, see why AdamW decouples weight decay, and walk a one-parameter numeric update step by step.
- How GPT Actually Works: A Visual Guide to Transformers How does a GPT actually work? A plain-English, visual walkthrough of transformers — tokens, word embeddings, why directions in vector space mean something, dot products, softmax and temperature — with the real GPT-3 numbers worked out.
- How Attention Works in Transformers: Queries, Keys and Values How does attention work in transformers? A visual, step-by-step explanation of the attention mechanism — queries, keys, values, the attention pattern, masking, the low-rank value trick, and multi-head attention — with the real GPT-3 parameter counts.
- Transformer from Scratch: The Full Forward Pass, Backprop, and Weight Update Math One full transformer training step worked by hand — embeddings, positional encoding, attention, layer norm, the encoder-decoder, cross-entropy loss, backprop, and Adam.
- GPT Math Explained: The Full Forward Pass Beyond Attention How does GPT math actually work end to end? Follow one training step from token IDs through Q/K/V, multi-head attention, MLP, logits, cross-entropy loss, backprop, and AdamW — past O = Attention(Q,K,V).
- Build GPT-2 from Scratch in PyTorch: A Full Walkthrough How to build GPT-2 from scratch in PyTorch — tokenization, causal self-attention, transformer blocks, weight tying, and a working training loop — implemented step by step and trained locally (124M parameters).
- DeepSeek V4 Claims Examined: Long-Context Engineering, Math, and Verification A verification-first analysis of circulating DeepSeek V4 claims, plus the real mathematics behind sparse attention, KV-cache scaling, stable residual mixing, and long-context training.
- Build a Mini LLM from Scratch in NumPy: RoPE, GQA, SwiGLU Visual Guide How to build a mini LLM from scratch in NumPy — RoPE, GQA, QK-Norm, SwiGLU, tied embeddings, and a 3.87M chat companion with full architecture visuals.