Build a Mini LLM from Scratch in NumPy: RoPE, GQA, SwiGLU
How to build a mini LLM from scratch in NumPy: RoPE, GQA, QK-Norm, SwiGLU, tied embeddings, and a 3.87M chat companion with full architecture visuals.
Cross-Entropy Loss Explained: From Logits to LLM Training
What is cross-entropy loss? Derive softmax and negative log-likelihood, work a 3-class example, and see why MSE fails for language model classification.
Adam and AdamW Explained: How LLMs Update Their Weights
How does the Adam optimizer work? Derive momentum, RMSprop and bias correction, see why AdamW decouples weight decay, and walk a numeric update by hand.
Token Embeddings Explained: How LLMs Turn IDs Into Vectors
What is a token embedding? How LLMs map token IDs to learned vectors: the embedding matrix, gather vs one-hot matmul, and a from-scratch NumPy build.
Positional Encoding Explained: How Transformers Learn Order
What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal embeddings, and a from-scratch NumPy implementation.
Byte Pair Encoding (BPE) Explained: How GPT Tokenizers Work
What is byte pair encoding (BPE)? How GPT tokenizers turn text into IDs: pretokenization, merge rules, byte-level vocab, and a from-scratch build.
Numerical Gradient Checking: Debug Your Autograd Engine
What is numerical gradient checking? The central difference formula, why you never train with finite differences, and a grad checker that catches real bugs.
Transformer from Scratch: Forward Pass and Backprop by Hand
One full transformer training step worked by hand: embeddings, positional encoding, attention, layer norm, cross-entropy loss, backprop, and Adam.
Build GPT-2 from Scratch in PyTorch: A Full Walkthrough
How to build GPT-2 from scratch in PyTorch: tokenization, causal self-attention, transformer blocks, weight tying, and a 124M training loop that runs.
Backpropagation from Scratch: Build an Autograd Engine
How does backpropagation work? Build a working autograd engine from scratch in ~80 lines of Python: computation graphs, chain rule, reverse-mode autodiff.