Token Embeddings Explained: How LLMs Turn IDs Into Vectors
What is a token embedding? How LLMs map token IDs to learned vectors: the embedding matrix, gather vs one-hot matmul, and a from-scratch NumPy build.
Every post tagged “Tokenization & Embeddings”, newest first.
What is a token embedding? How LLMs map token IDs to learned vectors: the embedding matrix, gather vs one-hot matmul, and a from-scratch NumPy build.
What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal embeddings, and a from-scratch NumPy implementation.
What is byte pair encoding (BPE)? How GPT tokenizers turn text into IDs: pretokenization, merge rules, byte-level vocab, and a from-scratch build.