All topics

Training & Optimization

Every post tagged “Training & Optimization”, newest first.

4 posts
Transformers 13 min

GPT Math Explained: The Full Forward Pass Beyond Attention

How does GPT math work end to end? One training step from token IDs through Q/K/V, attention, MLP, logits, cross-entropy loss, backprop, and AdamW.

Training & Optimization 11 min

Cross-Entropy Loss Explained: From Logits to LLM Training

What is cross-entropy loss? Derive softmax and negative log-likelihood, work a 3-class example, and see why MSE fails for language model classification.

Training & Optimization 13 min

Adam and AdamW Explained: How LLMs Update Their Weights

How does the Adam optimizer work? Derive momentum, RMSprop and bias correction, see why AdamW decouples weight decay, and walk a numeric update by hand.

LLMs 15 min

DeepSeek V4 Explained: Long-Context Engineering and Math

A verification-first look at DeepSeek V4 claims, plus the real math behind sparse attention, KV-cache scaling, and long-context training stability.