How does GPT math actually work end to end? Follow one training step from token IDs through Q/K/V, multi-head attention, MLP, logits, cross-entropy loss, backprop, and AdamW — past O = Attention(Q,K,V).
What is cross-entropy loss? Derive softmax + negative log-likelihood step by step, work a 3-class numeric example, see why MSE fails for classification, and build the loss GPT training actually uses.
How does the Adam optimizer work? Derive momentum, RMSprop, and bias correction, see why AdamW decouples weight decay, and walk a one-parameter numeric update step by step.
What is numerical gradient checking? Learn the central difference formula, why you never train with finite differences, and how to build a grad checker that catches bugs in your autograd before LayerNorm or attention.
One full transformer training step worked by hand — embeddings, positional encoding, attention, layer norm, the encoder-decoder, cross-entropy loss, backprop, and Adam.
How does attention work in transformers? A visual, step-by-step explanation of the attention mechanism — queries, keys, values, the attention pattern, masking, the low-rank value trick, and multi-head attention — with the real GPT-3 parameter counts.
How does a GPT actually work? A plain-English, visual walkthrough of transformers — tokens, word embeddings, why directions in vector space mean something, dot products, softmax and temperature — with the real GPT-3 numbers worked out.
How does backpropagation work? Build a working autograd engine from scratch in ~80 lines of Python — computation graphs, the chain rule, and reverse-mode autodiff — the same core idea behind PyTorch's .backward().