---
layout: post
title: "Byte Pair Encoding (BPE) Explained: How GPT Tokenizers Turn Text Into Numbers"
date: 2026-06-10
tags: ["Tokenization", "BPE", "GPT", "NLP", "From Scratch", "Python"]
category: "ai"
description: "What is byte pair encoding (BPE)? Learn how GPT tokenizers split text into IDs — pretokenization, merge rules, byte-level vocab, and a from-scratch implementation you can actually debug."
keywords: "byte pair encoding, BPE tokenization, how does BPE work, GPT tokenizer explained, train BPE from scratch, byte level BPE, pretokenization regex, tiktoken BPE"
image: /assets/images/og-byte-pair-encoding.png
order: 4
---

Before a transformer sees a single weight update, your text is already gone. Not metaphorically — literally converted into a list of integers. I used to treat tokenization as plumbing you import from `tiktoken` and forget. Then I started building [AL-1](/blog/build-autograd-from-scratch/), a GPT in pure NumPy, and realized the tokenizer is half the data pipeline. Get BPE wrong and your model trains on garbage IDs; get it right and suddenly `def foo():` compresses to a handful of tokens instead of twelve unrelated bytes.

Byte Pair Encoding is the algorithm behind GPT-2, GPT-3, LLaMA, and most modern LLM vocabularies. This post is one continuous walkthrough: why character-level and word-level tokenization fail, how byte-level BPE fixes both, the math of merge selection with every count written out, a complete training example on the tiny corpus `"low low lower lowest"` from pretokenization through four merges to final encodings, inference-time encoding step by step, decoding, and the Python I'm using in `zyn/bpe.py`. Same pipeline as [numerical gradient checking](/blog/numerical-gradient-checking-explained/) for autograd — own the data path before you trust the model.

```mermaid
flowchart LR
  TXT["Raw text\n'def hello():'"] --> PRE["Pretokenize\nregex splits"]
  PRE --> BYTES["Bytes per pretoken\n[100, 101, 102, ...]"]
  BYTES --> MERGE["Apply learned merges\nrank-ordered"]
  MERGE --> IDS["Token IDs\n[412, 891, 37]"]
  IDS --> EMB["Embedding lookup\nin the transformer"]
```

---

## Why can't the model just read characters?

You could map each Unicode character to an integer. Shakespeare at ~1 MB becomes ~1M tokens. A 1024 context window holds roughly one screen of text. Training becomes painfully slow because every forward pass processes far more positions than necessary.

Word-level tokenization has the opposite problem. The vocabulary explodes — every typo, hashtag, and `async def` variant is a new symbol you've never seen. Out-of-vocabulary (OOV) words become `<UNK>` and the model loses information.

BPE sits in the middle. Start with a small base vocabulary (256 raw bytes), then **learn** the most frequent pairs and merge them into new symbols. Common fragments like ` the`, `ing`, `def`, `()` get their own IDs. Rare strings still tokenize — they just fall back to bytes. No OOV dead ends.

---

## What is byte pair encoding?

BPE is a compression algorithm adapted for NLP. The training loop is brutally simple:

1. Split corpus into **pretokens** (word-like chunks).
2. Represent each pretoken as a sequence of byte IDs.
3. Count every adjacent pair $(a, b)$ across all pretokens, weighted by word frequency.
4. Merge the most frequent pair into a new token ID.
5. Repeat until you hit your target vocabulary size.

At inference time you don't recount anything. You greedily apply the merges you learned, in the order they were created (earliest merge = highest priority).

The original paper (Sennrich et al., 2016) operated on characters for neural machine translation. GPT-2 (Radford et al., 2019) went **byte-level**: each of the 256 UTF-8 bytes is a starting symbol. That single decision eliminates unknown Unicode — any string encodes to bytes, bytes decode back to UTF-8.

---

## Why bytes, not characters?

UTF-8 is variable-width. The character `é` might be one code point but two bytes. Byte-level BPE sidesteps the entire Unicode normalization rabbit hole:

- **Vocabulary floor is fixed:** 256 bytes + a few special tokens. You always know the minimum size.
- **No OOV:** `🤖`, Cyrillic, Zalgo text — all bytes, all representable.
- **Compression where it matters:** Frequent English/code fragments merge into multi-byte tokens; rare emoji stay as 4 byte tokens.

Tradeoff: byte sequences are longer than character sequences for non-Latin scripts. For English + Python code (what I'm training on), it's a good deal.

My implementation reserves **6 special tokens** before the 256 bytes:

| ID | Token | Purpose |
|---|---|---|
| 0 | `<pad>` | Batch padding |
| 1 | `<bos>` | Beginning of sequence |
| 2 | `<eos>` | End of sequence |
| 3–5 | `<tool_call>`, `</tool_call>`, `<tool_result>` | Tool-calling fine-tune later |

So `byte_base = 6`, and raw byte $b$ maps to token ID $6 + b$.

For the letter `l`:

$$
\text{ord}(\texttt{l}) = 108
$$

$$
\text{token\_id}(\texttt{l}) = 6 + 108 = 114
$$

For ASCII space:

$$
\text{ord}(\texttt{ }) = 32
$$

$$
\text{token\_id}(\texttt{ }) = 6 + 32 = 38
$$

Minimum vocabulary size:

$$
6 + 256 = 262
$$

You cannot train with `vocab_size < 262` — the code raises immediately.

---

## What is pretokenization?

BPE doesn't run on the raw string as one blob. First, a regex splits text into pretokens — chunks that look like words, numbers, or punctuation. GPT-2 uses this pattern (same in my code):

```python
PAT = re.compile(
    r"""'s|'t|'re|'ve|'m|'ll|'d| ?[^\W\d]+| ?\d+| ?[^\s\w]+|\s+(?!\S)|\s+"""
)
```

What it captures:

- Contractions: `'re`, `'ve`, `'ll`
- Words with optional leading space: ` hello`, `def`
- Numbers: ` 42`
- Punctuation clusters: `():`
- Whitespace runs

The leading-space convention matters. In English, a space before a word is semantically attached — GPT learns ` the` as one token, not ` ` + `the`. That's why `"low"` and `" low"` are different pretokens with different merge histories.

---

## How does training pick the next merge?

For each pretoken $w$ with frequency $\text{freq}(w)$, maintain a sequence of byte IDs. Count adjacent pairs:

$$
\text{count}(a, b) = \sum_{w} \text{freq}(w) \cdot \#\{(a,b) \text{ as consecutive IDs in } w\}
$$

Pick the pair with maximum count:

$$
(a^\*, b^\*) = \arg\max_{(a,b)} \text{count}(a, b)
$$

If the count is below `min_freq` (default 2), stop early.

Assign the merged pair a new ID `next_id`, store `merges[(a,b)] = next_id`, and set:

$$
\text{vocab}[\text{next\_id}] = \text{vocab}[a] \mathbin{\Vert} \text{vocab}[b]
$$

where $\Vert$ is byte concatenation. Then replace every occurrence of the pair $(a,b)$ with `next_id` in all pretoken sequences. Increment `next_id`. Loop until `next_id == vocab_size`.

```mermaid
flowchart TD
  A["Count all adjacent pairs\nweighted by pretoken freq"] --> B{"Best pair count\n≥ min_freq?"}
  B -->|no| STOP["Stop training"]
  B -->|yes| C["Assign new ID,\nstore merge rule"]
  C --> D["Concat vocab bytes\nvocab[new] = vocab[a]+vocab[b]"]
  D --> E["Replace pair everywhere\nin pretoken ID lists"]
  E --> F{"next_id < vocab_size?"}
  F -->|yes| A
  F -->|no| STOP
```

---

## Full worked example: `"low low lower lowest"`

I will train BPE on the corpus `"low low lower lowest"` with `byte_base = 6`, `vocab_size = 280`, and `min_freq = 2`. Every pretokenization step, byte conversion, pair count, merge, and final encoding is written on its own line. This is the same worked example GPT tutorials use — I'm using real token IDs from my implementation, not abstract letters.

### Step 1 — Pretokenize the corpus

Input string:

$$
\text{corpus} = \texttt{"low low lower lowest"}
$$

The regex `PAT.findall` splits on word boundaries and keeps leading spaces attached:

$$
\text{pretokens} = [\texttt{'low'}, \texttt{' low'}, \texttt{' lower'}, \texttt{' lowest'}]
$$

Each distinct pretoken appears once in this tiny corpus:

$$
\text{freq}(\texttt{'low'}) = 1
$$

$$
\text{freq}(\texttt{' low'}) = 1
$$

$$
\text{freq}(\texttt{' lower'}) = 1
$$

$$
\text{freq}(\texttt{' lowest'}) = 1
$$

Note: `"low"` and `" low"` are different strings because of the leading space on the second occurrence.

### Step 2 — Convert each pretoken to byte token IDs

Token ID formula: $\text{id} = 6 + \text{byte}$.

**Pretoken `'low'`** — bytes `l`, `o`, `w`:

$$
\text{ord}(\texttt{l}) = 108 \quad \Rightarrow \quad 6 + 108 = 114
$$

$$
\text{ord}(\texttt{o}) = 111 \quad \Rightarrow \quad 6 + 111 = 117
$$

$$
\text{ord}(\texttt{w}) = 119 \quad \Rightarrow \quad 6 + 119 = 125
$$

$$
\text{word\_ids}[\texttt{'low'}] = [114, 117, 125]
$$

**Pretoken `' low'`** — bytes space, `l`, `o`, `w`:

$$
\text{ord}(\texttt{ }) = 32 \quad \Rightarrow \quad 6 + 32 = 38
$$

$$
\text{word\_ids}[\texttt{' low'}] = [38, 114, 117, 125]
$$

**Pretoken `' lower'`** — bytes space, `l`, `o`, `w`, `e`, `r`:

$$
\text{ord}(\texttt{e}) = 101 \quad \Rightarrow \quad 6 + 101 = 107
$$

$$
\text{ord}(\texttt{r}) = 114 \quad \Rightarrow \quad 6 + 114 = 120
$$

$$
\text{word\_ids}[\texttt{' lower'}] = [38, 114, 117, 125, 107, 120]
$$

**Pretoken `' lowest'`** — bytes space, `l`, `o`, `w`, `e`, `s`, `t`:

$$
\text{ord}(\texttt{s}) = 115 \quad \Rightarrow \quad 6 + 115 = 121
$$

$$
\text{ord}(\texttt{t}) = 116 \quad \Rightarrow \quad 6 + 116 = 122
$$

$$
\text{word\_ids}[\texttt{' lowest'}] = [38, 114, 117, 125, 107, 121, 122]
$$

### Step 3 — Count all adjacent pairs (before any merges)

Pair $(114, 117)$ — the bytes `lo` — appears in all four pretokens:

$$
\#\text{ in }\texttt{'low'} = 1
$$

$$
\#\text{ in }\texttt{' low'} = 1
$$

$$
\#\text{ in }\texttt{' lower'} = 1
$$

$$
\#\text{ in }\texttt{' lowest'} = 1
$$

$$
\text{count}(114, 117) = 1 + 1 + 1 + 1 = 4
$$

Pair $(117, 125)$ — the bytes `ow` — same four pretokens:

$$
\text{count}(117, 125) = 4
$$

Pair $(38, 114)$ — space followed by `l`:

$$
\#\text{ in }\texttt{' low'} = 1
$$

$$
\#\text{ in }\texttt{' lower'} = 1
$$

$$
\#\text{ in }\texttt{' lowest'} = 1
$$

$$
\text{count}(38, 114) = 3
$$

Pair $(125, 107)$ — `we` — only in ` lower` and ` lowest`:

$$
\text{count}(125, 107) = 2
$$

All other pairs have count $1$. The maximum is $4$. Ties go to whichever pair the `Counter` returns first — here $(114, 117)$ wins.

### Step 4 — Merge 1: $(114, 117) \rightarrow 262$

New token ID:

$$
\text{next\_id} = 262
$$

Vocabulary entry:

$$
\text{vocab}[262] = \text{vocab}[114] \mathbin{\Vert} \text{vocab}[117] = b\texttt{'lo'}
$$

Replace $(114, 117)$ everywhere:

$$
\text{word\_ids}[\texttt{'low'}] = [262, 125]
$$

$$
\text{word\_ids}[\texttt{' low'}] = [38, 262, 125]
$$

$$
\text{word\_ids}[\texttt{' lower'}] = [38, 262, 125, 107, 120]
$$

$$
\text{word\_ids}[\texttt{' lowest'}] = [38, 262, 125, 107, 121, 122]
$$

$$
\text{next\_id} = 263
$$

### Step 5 — Count pairs after merge 1

Pair $(262, 125)$ — merged `lo` followed by `w` — in all four pretokens:

$$
\text{count}(262, 125) = 4
$$

Pair $(38, 262)$ — space followed by merged `lo`:

$$
\text{count}(38, 262) = 3
$$

Pair $(125, 107)$ — `we`:

$$
\text{count}(125, 107) = 2
$$

Maximum count is $4$. Merge $(262, 125)$.

### Step 6 — Merge 2: $(262, 125) \rightarrow 263$

$$
\text{vocab}[263] = \text{vocab}[262] \mathbin{\Vert} \text{vocab}[125] = b\texttt{'low'}
$$

Replace $(262, 125)$ everywhere:

$$
\text{word\_ids}[\texttt{'low'}] = [263]
$$

$$
\text{word\_ids}[\texttt{' low'}] = [38, 263]
$$

$$
\text{word\_ids}[\texttt{' lower'}] = [38, 263, 107, 120]
$$

$$
\text{word\_ids}[\texttt{' lowest'}] = [38, 263, 107, 121, 122]
$$

$$
\text{next\_id} = 264
$$

The pretoken `'low'` is now a **single token**.

### Step 7 — Count pairs after merge 2

Pair $(38, 263)$ — space + `low` — in three pretokens (` low`, ` lower`, ` lowest`):

$$
\text{count}(38, 263) = 3
$$

Pair $(263, 107)$ — `low` + `e`:

$$
\text{count}(263, 107) = 2
$$

Maximum is $3$. Merge $(38, 263)$.

### Step 8 — Merge 3: $(38, 263) \rightarrow 264$

$$
\text{vocab}[264] = \text{vocab}[38] \mathbin{\Vert} \text{vocab}[263] = b\texttt{' low'}
$$

Replace $(38, 263)$ everywhere:

$$
\text{word\_ids}[\texttt{'low'}] = [263]
$$

$$
\text{word\_ids}[\texttt{' low'}] = [264]
$$

$$
\text{word\_ids}[\texttt{' lower'}] = [264, 107, 120]
$$

$$
\text{word\_ids}[\texttt{' lowest'}] = [264, 107, 121, 122]
$$

$$
\text{next\_id} = 265
$$

### Step 9 — Count pairs after merge 3

Pair $(264, 107)$ — ` low` + `e` — in ` lower` and ` lowest`:

$$
\text{count}(264, 107) = 2
$$

This meets `min_freq = 2`. Merge it.

### Step 10 — Merge 4: $(264, 107) \rightarrow 265$

$$
\text{vocab}[265] = \text{vocab}[264] \mathbin{\Vert} \text{vocab}[107] = b\texttt{' lowe'}
$$

Replace $(264, 107)$ everywhere:

$$
\text{word\_ids}[\texttt{'low'}] = [263]
$$

$$
\text{word\_ids}[\texttt{' low'}] = [264]
$$

$$
\text{word\_ids}[\texttt{' lower'}] = [265, 120]
$$

$$
\text{word\_ids}[\texttt{' lowest'}] = [265, 121, 122]
$$

$$
\text{next\_id} = 266
$$

### Step 11 — Final encodings after training

| Pretoken | Token IDs | Decoded bytes |
|---|---|---|
| `'low'` | `[263]` | `low` |
| `' low'` | `[264]` | ` low` |
| `' lower'` | `[265, 120]` | ` low` + `e` + `r` |
| `' lowest'` | `[265, 121, 122]` | ` lowe` + `s` + `t` |

The shared prefix ` lowe` merged once into token 265. The suffix `r` in ` lower` and `st` in ` lowest` stay as individual byte tokens because they appeared only once. That is the compression story — common substrings collapse, rare suffixes don't.

Merge rank table (encoding priority):

$$
\text{ranks}[(114, 117)] = 0
$$

$$
\text{ranks}[(262, 125)] = 1
$$

$$
\text{ranks}[(38, 263)] = 2
$$

$$
\text{ranks}[(264, 107)] = 3
$$

### Step 12 — Encode `'lowest'` at inference time

The string `'lowest'` has no leading space, so it pretokenizes to a single chunk `'lowest'` (not in our training corpus, but the learned merges still apply).

Convert to byte IDs:

$$
\text{ord}(\texttt{l}) = 108 \quad \Rightarrow \quad 114
$$

$$
\text{ord}(\texttt{o}) = 111 \quad \Rightarrow \quad 117
$$

$$
\text{ord}(\texttt{w}) = 119 \quad \Rightarrow \quad 125
$$

$$
\text{ord}(\texttt{e}) = 101 \quad \Rightarrow \quad 107
$$

$$
\text{ord}(\texttt{s}) = 115 \quad \Rightarrow \quad 121
$$

$$
\text{ord}(\texttt{t}) = 116 \quad \Rightarrow \quad 122
$$

$$
\text{ids} = [114, 117, 125, 107, 121, 122]
$$

**Greedy merge pass 1** — scan adjacent pairs, pick lowest rank:

$$
(114, 117) \text{ has rank } 0
$$

$$
(117, 125) \text{ has no merge}
$$

$$
(125, 107) \text{ has no merge}
$$

$$
\vdots
$$

Best pair: $(114, 117)$ with rank $0$. Apply merge $\rightarrow 262$:

$$
\text{ids} = [262, 125, 107, 121, 122]
$$

**Greedy merge pass 2**:

$$
(262, 125) \text{ has rank } 1
$$

All other adjacent pairs have no merge rule. Apply merge $\rightarrow 263$:

$$
\text{ids} = [263, 107, 121, 122]
$$

**Greedy merge pass 3** — no adjacent pair exists in `merges`. Stop.

Final encoding:

$$
\text{encode}(\texttt{'lowest'}) = [263, 107, 121, 122]
$$

One merged `low` token plus three byte tokens for `est`.

With BOS/EOS:

```python
t.encode("lowest", add_bos=True, add_eos=True)
# [1, 263, 107, 121, 122, 2]
```

Round-trip check:

$$
\text{decode}([263, 107, 121, 122]) = \texttt{'lowest'}
$$

---

## How does decoding work?

Decoding is the easy direction. Each token ID maps to a byte string in `vocab`. Concatenate, decode UTF-8:

$$
\text{text} = \text{UTF-8}^{-1}\left(\mathop{\Big\Vert}_{i}\ \text{vocab}[\text{id}_i]\right)
$$

For our encoding $[263, 107, 121, 122]$:

$$
\text{vocab}[263] = b\texttt{'low'}
$$

$$
\text{vocab}[107] = b\texttt{'e'}
$$

$$
\text{vocab}[121] = b\texttt{'s'}
$$

$$
\text{vocab}[122] = b\texttt{'t'}
$$

$$
b\texttt{'low'} \mathbin{\Vert} b\texttt{'e'} \mathbin{\Vert} b\texttt{'s'} \mathbin{\Vert} b\texttt{'t'} = b\texttt{'lowest'}
$$

$$
\text{UTF-8}^{-1}(b\texttt{'lowest'}) = \texttt{'lowest'}
$$

Special tokens can be skipped with `skip_specials=True` so `<bos>` doesn't appear in generated text.

The invariant I test before wiring the dataloader:

$$
\operatorname{decode}(\operatorname{encode}(\text{text})) = \text{text}
$$

---

## The implementation skeleton

The full tokenizer lives in ~160 lines. Four pieces do the real work.

**Pretokenize and bytes:**

```python
def _pretokens(text: str) -> list[str]:
    return PAT.findall(text)

def _word_to_ids(word: str, byte_base: int) -> list[int]:
    return [byte_base + b for b in word.encode("utf-8")]
```

**Merge a pair in a sequence** (left-to-right, non-overlapping):

```python
def _merge_seq(ids: list[int], pair: tuple[int, int], new_id: int) -> list[int]:
    a, b = pair
    out, i, n = [], 0, len(ids)
    while i < n:
        if i < n - 1 and ids[i] == a and ids[i + 1] == b:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out
```

**Training loop** — count pairs, pick best, merge everywhere:

```python
while next_id < vocab_size:
    pair_counts = Counter()
    for w, freq in word_freq.items():
        for a, b in zip(word_ids[w], word_ids[w][1:]):
            pair_counts[(a, b)] += freq
    best, cnt = pair_counts.most_common(1)[0]
    if cnt < min_freq:
        break
    self.merges[best] = next_id
    self.vocab[next_id] = self.vocab[best[0]] + self.vocab[best[1]]
    for w in word_ids:
        word_ids[w] = _merge_seq(word_ids[w], best, next_id)
    next_id += 1
self.ranks = {pair: i for i, pair in enumerate(self.merges)}
```

**Encoding** — greedy lowest-rank merge:

```python
while len(ids) >= 2:
    best_pair, best_rank = None, None
    for pair in zip(ids, ids[1:]):
        r = self.ranks.get(pair)
        if r is not None and (best_rank is None or r < best_rank):
            best_rank, best_pair = r, pair
    if best_pair is None:
        break
    ids = _merge_seq(ids, best_pair, self.merges[best_pair])
```

Save/load serializes `specials` and the merge list to JSON — no pickle, easy to inspect.

```mermaid
sequenceDiagram
  participant T as Text
  participant P as Pretokenizer
  participant B as Byte IDs
  participant M as Merge engine
  participant V as Vocab lookup

  T->>P: findall(regex)
  P->>B: UTF-8 encode each pretoken
  B->>M: greedy rank merges
  M->>V: final token IDs
  Note over V: decode = concat bytes → UTF-8
```

---

## What vocabulary size should you use?

For my AL-1 code model I'm targeting **8k–16k** merges on top of the 262-byte floor. Code has more distinct symbols than TinyShakespeare — operators, indentation, string delimiters. A 4k vocab works for prose; Python benefits from headroom.

Training data: I'm using a 100-file slice of [the-stack-smol-xs](https://huggingface.co/datasets/bigcode/the-stack-smol-xs) Python split (~1 MB) for smoke tests, then scaling to a ~10 MB sample for the real merge table. You don't need the full corpus to learn reasonable merges — frequency tails off fast.

| Setting | My value | Why |
|---|---|---|
| `vocab_size` | 8,000–16,000 | Code needs more symbols than char-level |
| `min_freq` | 2 | Ignore one-off pair noise |
| Special tokens | 6 | Pad, boundaries, future tool calls |
| Floor | 262 | $6 + 256$ bytes |

Compare to GPT-2: **50,257** tokens. Same algorithm, bigger corpus, more merges.

Tokenizer design is also a model-efficiency trade. For vocabulary size $V$ and model width $d$, an untied input plus output embedding table costs about $2Vd$ parameters. Attention cost grows roughly with sequence length squared. A smaller vocabulary means fewer embedding parameters but longer sequences; a larger vocabulary shortens sequences but inflates the embedding matrices and creates rarer tokens that receive fewer gradient updates.

---

## BPE vs WordPiece vs SentencePiece

| Method | Merge criterion | Used by |
|---|---|---|
| **BPE** | Most frequent pair | GPT-2/3/4, LLaMA, Mistral |
| **WordPiece** | Pair that maximizes language-model likelihood | BERT |
| **SentencePiece** | BPE or unigram, trained on raw bytes with meta-symbols | T5, LLaMA 2 tokenizer option |

For a decoder-only GPT clone, BPE is the path of least resistance — it's what [tiktoken](https://github.com/openai/tiktoken) implements for `gpt2` encoding, and the merge-rank encode logic matches what you need for a from-scratch rebuild.

---

## Common bugs I watch for

**Forgetting the leading space in pretokens.** `"hello"` and `" hello"` learn different merges. If your training data strips spaces but inference doesn't, token counts drift.

**Off-by-six in byte IDs.** Special tokens shift everything. `ord('a')` is 97, but the token ID is 103 when `byte_base=6`. Every pair count depends on this.

**Merge order mismatch.** Encoding must use `ranks` (creation order), not frequency at encode time. Re-sorting merges breaks round-trip.

**`vocab_size` too small.** Below 262 the code refuses to start. Below ~1000 on code and you're mostly bytes anyway — defeats the purpose.

Run round-trip tests on snippets from your actual corpus before you wire the tokenizer into the dataloader. Same advice as [numerical gradient checking](/blog/numerical-gradient-checking-explained/) for autograd — catch data-pipeline bugs before they poison training.

---

## Where this fits in the build

AL-1 pipeline so far: [autograd engine](/blog/build-autograd-from-scratch/) → [gradient checker](/blog/numerical-gradient-checking-explained/) → **BPE tokenizer** → [token embeddings](/blog/token-embeddings-explained/) → [positional encoding](/blog/positional-encoding-explained/) → transformer blocks.

The tokenizer is the bridge between raw `.py` files and the integer sequences the embedding layer consumes. The [GPT-2 architecture post](/blog/build-gpt2-from-scratch/) shows where those IDs go once they're ready — `wte`, causal attention, `lm_head` with weight tying.

I built this tokenizer by hand because I wanted to stare at merge tables when generation looks wrong. When `import tiktoken` is one line, you lose visibility into whether the bug is the model or the bytes going in. For a toy GPT that might be overkill. For learning how these systems actually work, it's the difference between using a library and owning the pipeline.

Train the merges. Check round-trips. Then trust your IDs.