How does GPT math actually work end to end? Follow one training step from token IDs through Q/K/V, multi-head attention, MLP, logits, cross-entropy loss, backprop, and AdamW — past O = Attention(Q,K,V).
What is cross-entropy loss? Derive softmax + negative log-likelihood step by step, work a 3-class numeric example, see why MSE fails for classification, and build the loss GPT training actually uses.
How does the Adam optimizer work? Derive momentum, RMSprop, and bias correction, see why AdamW decouples weight decay, and walk a one-parameter numeric update step by step.
What is a token embedding? How LLMs map token IDs to learned vectors — the embedding matrix, gather lookup vs one-hot matmul, parameter counts, and a from-scratch NumPy implementation.
What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal position embeddings, why we add instead of concatenate, and a from-scratch NumPy implementation.
What is byte pair encoding (BPE)? Learn how GPT tokenizers split text into IDs — pretokenization, merge rules, byte-level vocab, and a from-scratch implementation you can actually debug.
How does a GPT actually work? A plain-English, visual walkthrough of transformers — tokens, word embeddings, why directions in vector space mean something, dot products, softmax and temperature — with the real GPT-3 numbers worked out.