All topics

Tokenization & Embeddings

Every post tagged “Tokenization & Embeddings”, newest first.

3 posts
Tokenization & Embeddings 17 min

Token Embeddings Explained: How LLMs Turn IDs Into Vectors

What is a token embedding? How LLMs map token IDs to learned vectors: the embedding matrix, gather vs one-hot matmul, and a from-scratch NumPy build.

Tokenization & Embeddings 14 min

Positional Encoding Explained: How Transformers Learn Order

What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal embeddings, and a from-scratch NumPy implementation.

Tokenization & Embeddings 23 min

Byte Pair Encoding (BPE) Explained: How GPT Tokenizers Work

What is byte pair encoding (BPE)? How GPT tokenizers turn text into IDs: pretokenization, merge rules, byte-level vocab, and a from-scratch build.