All topics

AdamW

Every post tagged “AdamW”, newest first.

2 posts
GPT 10 min

GPT Math Explained: The Full Forward Pass Beyond Attention

How does GPT math actually work end to end? Follow one training step from token IDs through Q/K/V, multi-head attention, MLP, logits, cross-entropy loss, backprop, and AdamW — past O = Attention(Q,K,V).
Deep Learning 11 min

Adam and AdamW Explained: How Modern LLMs Actually Update Their Weights

How does the Adam optimizer work? Derive momentum, RMSprop, and bias correction, see why AdamW decouples weight decay, and walk a one-parameter numeric update step by step.