All topics

Python

Every post tagged “Python”, newest first.

7 posts
Deep Learning 9 min

Cross-Entropy Loss Explained: From Logits to Language Model Training

What is cross-entropy loss? Derive softmax + negative log-likelihood step by step, work a 3-class numeric example, see why MSE fails for classification, and build the loss GPT training actually uses.
Deep Learning 11 min

Adam and AdamW Explained: How Modern LLMs Actually Update Their Weights

How does the Adam optimizer work? Derive momentum, RMSprop, and bias correction, see why AdamW decouples weight decay, and walk a one-parameter numeric update step by step.
Embeddings 15 min

Token Embeddings Explained: How LLMs Turn Token IDs Into Meaning Vectors

What is a token embedding? How LLMs map token IDs to learned vectors — the embedding matrix, gather lookup vs one-hot matmul, parameter counts, and a from-scratch NumPy implementation.
Positional Encoding 12 min

Positional Encoding Explained: How Transformers Learn Word Order

What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal position embeddings, why we add instead of concatenate, and a from-scratch NumPy implementation.
Tokenization 21 min

Byte Pair Encoding (BPE) Explained: How GPT Tokenizers Turn Text Into Numbers

What is byte pair encoding (BPE)? Learn how GPT tokenizers split text into IDs — pretokenization, merge rules, byte-level vocab, and a from-scratch implementation you can actually debug.
Deep Learning 14 min

Numerical Gradient Checking: How to Debug Your Autograd Engine Before Training GPT

What is numerical gradient checking? Learn the central difference formula, why you never train with finite differences, and how to build a grad checker that catches bugs in your autograd before LayerNorm or attention.
Deep Learning 31 min

Backpropagation from Scratch: Build an Autograd Engine in Python

How does backpropagation work? Build a working autograd engine from scratch in ~80 lines of Python — computation graphs, the chain rule, and reverse-mode autodiff — the same core idea behind PyTorch's .backward().