All topics

BPE

Every post tagged “BPE”, newest first.

1 post
Tokenization 21 min

Byte Pair Encoding (BPE) Explained: How GPT Tokenizers Work

What is byte pair encoding (BPE)? How GPT tokenizers turn text into IDs: pretokenization, merge rules, byte-level vocab, and a from-scratch build.