---
layout: post
title: "Positional Encoding Explained: How Transformers Learn Word Order"
date: 2026-06-10
tags: ["Positional Encoding", "Embeddings", "Transformers", "GPT", "From Scratch", "Python"]
category: "ai"
description: "What is positional encoding in transformers? Why attention is order-blind, learned vs sinusoidal position embeddings, why we add instead of concatenate, and a from-scratch NumPy implementation."
keywords: "positional encoding, positional embeddings explained, transformer word order, learned vs sinusoidal positional encoding, why add positional embedding, position embedding matrix, wpe GPT-2"
image: /assets/images/og-positional-encoding.png
order: 6
---

I closed the [token embeddings post](/blog/token-embeddings-explained/) with a problem I'd waved away: the embedding lookup is order-blind. `gather` returns the same vector for `cat` whether it's the first word or the hundredth. And attention — the thing those vectors flow into — is *permutation-invariant*. Shuffle the tokens and, without help, the math produces the same set of outputs in shuffled order. A transformer is, by default, a bag-of-words machine. That broke my mental model the first time I really sat with it, because order is most of language.

$$
\text{"Dog bites man"} \neq \text{"Man bites dog"}.
$$

Same three words. Same three token embeddings. Opposite meaning. Something has to tell the model *where* each token sits. That something is positional encoding. In GPT it's a second lookup table living right next to the [token embedding](/blog/token-embeddings-explained/) matrix. I'll walk through why attention can't see order on its own, what positional vectors actually are, the math of adding them, a full three-word worked example with every addition written out, learned vs sinusoidal encodings with formula steps, and where this lands in my [AL-1](/blog/build-autograd-from-scratch/) build.

---

## Why is attention order-blind?

Self-attention computes a weighted sum over all tokens. The weights come from dot products between query and key vectors. A dot product is just

$$
q_i \cdot k_j = \sum_{k=1}^{d} q_{i,k}\, k_{j,k}.
$$

There is no position index $t$ inside that sum. The number $q_i \cdot k_j$ depends only on the *vectors*, not on whether $i$ is slot 0 or slot 99. Shuffle the tokens and you shuffle the same vectors into different slots — attention still computes the same pairwise dot products, just permuted. Without extra information, the model genuinely cannot tell position 0 from position 100.

Picture six people in a room — Alice, Bob, Charlie, David, Eva, Frank. You're told *who* is present but not *where* they're sitting. That's a transformer with token embeddings alone: it knows the cast, not the seating chart. Language needs the seating chart.

```mermaid
flowchart LR
  subgraph blind["Token embeddings only"]
    a1["cat @ pos 0"] --> v1["vector V"]
    a2["cat @ pos 99"] --> v1
  end
  subgraph fixed["+ Positional encoding"]
    b1["cat @ pos 0"] --> w1["V + P0"]
    b2["cat @ pos 99"] --> w2["V + P99"]
  end
```

Same word `cat`, same embedding vector $V$. Only after adding a position-specific vector $P_t$ does slot 0 become $V + P_0$ and slot 99 become $V + P_{99}$ — two different inputs to attention.

---

## What is a positional embedding?

The intuition is a second lookup table, parallel to the token table. Token embedding answers "what word is this?" Positional embedding answers "where does it sit?"

Stack all position vectors into a matrix

$$
P \in \mathbb{R}^{L \times d}
$$

where $L$ is the maximum sequence length and $d$ is `d_model`. Row $t$ is the vector that means "I am the token at position $t$."

GPT-2 small uses

$$
L = 1024
$$

$$
d = 768
$$

$$
P \in \mathbb{R}^{1024 \times 768}
$$

$$
1024 \times 768 = 786{,}432 \text{ parameters}
$$

Roughly 800k parameters whose only job is to encode order. It's the same `gather` lookup as token embeddings, indexed by position ID instead of token ID.

Toy table with $L = 5$, $d = 2$:

$$
P =
\begin{bmatrix}
0.1 & 0.2 \\
0.3 & 0.4 \\
0.5 & 0.6 \\
0.7 & 0.8 \\
0.9 & 1.0
\end{bmatrix}
$$

$$
\text{row } 0 = \text{first token}
$$

$$
\text{row } 1 = \text{second token}
$$

$$
\vdots
$$

The implementation is two lines longer than the embedding layer, because the indices are generated, not passed in:

```python
class PositionalEmbedding:
    def __init__(self, max_seq, d_model, std=0.02):
        w = np.random.randn(max_seq, d_model) * std
        self.weight = Tensor(w)              # (max_seq, d_model), trainable

    def forward(self, seq_len):
        positions = np.arange(seq_len)       # [0, 1, 2, ..., seq_len-1]
        return self.weight.gather(positions, dim=0)   # (seq_len, d_model)
```

For a 5-token sentence, `np.arange(5)` makes `[0, 1, 2, 3, 4]` and `gather` pulls those five rows. Same $0.02$ GPT-2 initializer as the token embeddings — start small so training isn't fighting huge activations on step one. Cost is $O(L)$ row selection, no matmul. Next to attention's $O(L^2)$, positional lookup is rounding error.

---

## The math: why we add, not concatenate

The transformer input is the **sum** of the two tables, elementwise:

$$
X = E + P
$$

where $E$ holds token embeddings and $P$ holds position embeddings, both with shape $(L, d)$.

Take a toy sentence of three tokens with $d = 2$:

$$
E =
\begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
$$

$$
P =
\begin{bmatrix}
0.1 & 0.2 \\
0.3 & 0.4 \\
0.5 & 0.6
\end{bmatrix}
$$

Row 0 of $X$:

$$
X_{0,0} = E_{0,0} + P_{0,0}
$$

$$
X_{0,0} = 1 + 0.1
$$

$$
X_{0,0} = 1.1
$$

$$
X_{0,1} = E_{0,1} + P_{0,1}
$$

$$
X_{0,1} = 2 + 0.2
$$

$$
X_{0,1} = 2.2
$$

Row 1 of $X$:

$$
X_{1,0} = 3 + 0.3 = 3.3
$$

$$
X_{1,1} = 4 + 0.4 = 4.4
$$

Row 2 of $X$:

$$
X_{2,0} = 5 + 0.5 = 5.5
$$

$$
X_{2,1} = 6 + 0.6 = 6.6
$$

So

$$
X = E + P =
\begin{bmatrix}
1.1 & 2.2 \\
3.3 & 4.4 \\
5.5 & 6.6
\end{bmatrix}.
$$

That tiny addition is the whole trick. After it, `cat` at position 0 and `cat` at position 99 are genuinely different vectors going into attention.

Why add two unrelated things instead of concatenating? Concatenation would stick $[1, 2]$ and $[0.1, 0.2]$ into $[1, 2, 0.1, 0.2]$, doubling $d$. Attention's projection cost scales with $d^2$, so doubling width quadruples that cost and inflates every downstream weight. Addition keeps $d$ fixed — free position information, zero width tax. The model has 768 dimensions to play with; gradient descent learns to carve out subspaces where token identity and position don't collide. The sum stays disentangleable enough for attention to read both.

Like token embeddings, $P$ starts as scaled noise. Backprop only updates the positions a batch actually used — a 5-token sequence sends gradients into rows 0–4 and leaves rows 5–1023 alone that step. Over training the rows specialize. Nobody hand-codes any of it; it falls out of next-token prediction, same as everything else.

---

## Worked example: "dog bites man" vs "man bites dog"

I'll use $d = 2$ so every addition fits on one line. Three token embeddings from the [token embedding](/blog/token-embeddings-explained/) table:

$$
e_{\text{dog}} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}
$$

$$
e_{\text{bites}} = \begin{bmatrix} 3 \\ 1 \end{bmatrix}
$$

$$
e_{\text{man}} = \begin{bmatrix} 0 \\ 3 \end{bmatrix}
$$

Three learned position rows:

$$
p_0 = \begin{bmatrix} 0.1 \\ 0.0 \end{bmatrix}
$$

$$
p_1 = \begin{bmatrix} 0.0 \\ 0.2 \end{bmatrix}
$$

$$
p_2 = \begin{bmatrix} 0.3 \\ 0.1 \end{bmatrix}
$$

### Sentence 1: "dog bites man"

Token at position 0 is `dog`. Combine embedding with position 0:

$$
x_0 = e_{\text{dog}} + p_0
$$

$$
x_{0,0} = 1 + 0.1
$$

$$
x_{0,0} = 1.1
$$

$$
x_{0,1} = 2 + 0.0
$$

$$
x_{0,1} = 2.0
$$

$$
x_0 = \begin{bmatrix} 1.1 \\ 2.0 \end{bmatrix}
$$

Token at position 1 is `bites`:

$$
x_1 = e_{\text{bites}} + p_1
$$

$$
x_{1,0} = 3 + 0.0
$$

$$
x_{1,0} = 3.0
$$

$$
x_{1,1} = 1 + 0.2
$$

$$
x_{1,1} = 1.2
$$

$$
x_1 = \begin{bmatrix} 3.0 \\ 1.2 \end{bmatrix}
$$

Token at position 2 is `man`:

$$
x_2 = e_{\text{man}} + p_2
$$

$$
x_{2,0} = 0 + 0.3
$$

$$
x_{2,0} = 0.3
$$

$$
x_{2,1} = 3 + 0.1
$$

$$
x_{2,1} = 3.1
$$

$$
x_2 = \begin{bmatrix} 0.3 \\ 3.1 \end{bmatrix}
$$

Full input matrix for "dog bites man":

$$
X_{\text{dbm}} =
\begin{bmatrix}
1.1 & 2.0 \\
3.0 & 1.2 \\
0.3 & 3.1
\end{bmatrix}
$$

### Sentence 2: "man bites dog"

Same three token embeddings. Same three position rows. Different order — so each token meets a different position vector.

Token at position 0 is `man`:

$$
x'_0 = e_{\text{man}} + p_0
$$

$$
x'_{0,0} = 0 + 0.1
$$

$$
x'_{0,0} = 0.1
$$

$$
x'_{0,1} = 3 + 0.0
$$

$$
x'_{0,1} = 3.0
$$

$$
x'_0 = \begin{bmatrix} 0.1 \\ 3.0 \end{bmatrix}
$$

Token at position 1 is `bites`:

$$
x'_1 = e_{\text{bites}} + p_1
$$

$$
x'_{1,0} = 3 + 0.0
$$

$$
x'_{1,0} = 3.0
$$

$$
x'_{1,1} = 1 + 0.2
$$

$$
x'_{1,1} = 1.2
$$

$$
x'_1 = \begin{bmatrix} 3.0 \\ 1.2 \end{bmatrix}
$$

Token at position 2 is `dog`:

$$
x'_2 = e_{\text{dog}} + p_2
$$

$$
x'_{2,0} = 1 + 0.3
$$

$$
x'_{2,0} = 1.3
$$

$$
x'_{2,1} = 2 + 0.1
$$

$$
x'_{2,1} = 2.1
$$

$$
x'_2 = \begin{bmatrix} 1.3 \\ 2.1 \end{bmatrix}
$$

Full input matrix for "man bites dog":

$$
X_{\text{mbd}} =
\begin{bmatrix}
0.1 & 3.0 \\
3.0 & 1.2 \\
1.3 & 2.1
\end{bmatrix}
$$

### What changed?

Compare row by row. Position 0 used to be `dog` at $[1.1, 2.0]$; now it's `man` at $[0.1, 3.0]$. Position 2 used to be `man` at $[0.3, 3.1]$; now it's `dog` at $[1.3, 2.1]$. Only position 1 (`bites`) lands on the same combined vector $[3.0, 1.2]$ — because `bites` stayed in the middle.

Without positional encoding, sentence 1 would be $[e_{\text{dog}}, e_{\text{bites}}, e_{\text{man}}]$ and sentence 2 would be $[e_{\text{man}}, e_{\text{bites}}, e_{\text{dog}}]$ — a permutation of the same three vectors. Attention is blind to that shuffle. With $P$ added, $X_{\text{dbm}} \neq X_{\text{mbd}}$ as matrices. Query-key dot products inside attention now produce different scores. The model can finally tell who did the biting.

---

## Learned vs sinusoidal: the design fork

The original *Attention Is All You Need* paper used **fixed sinusoidal** encodings — no learned parameters, computed from a formula. For position $t$ and dimension index $i$ (with model width $d$):

$$
PE(t, 2i) = \sin\!\left(\frac{t}{10000^{2i/d}}\right)
$$

$$
PE(t, 2i+1) = \cos\!\left(\frac{t}{10000^{2i/d}}\right)
$$

Even dimensions get sines. Odd dimensions get cosines. Each dimension oscillates at a different wavelength because the divisor $10000^{2i/d}$ grows with $i$.

Worked numbers for $t = 1$, $i = 0$, $d = 4$:

$$
\frac{2i}{d} = \frac{0}{4} = 0
$$

$$
10000^{0} = 1
$$

$$
\frac{t}{10000^{2i/d}} = \frac{1}{1} = 1
$$

$$
PE(1, 0) = \sin(1) \approx 0.8415
$$

$$
PE(1, 1) = \cos(1) \approx 0.5403
$$

For $t = 1$, $i = 1$:

$$
\frac{2i}{d} = \frac{2}{4} = 0.5
$$

$$
10000^{0.5} = 100
$$

$$
\frac{t}{10000^{2i/d}} = \frac{1}{100} = 0.01
$$

$$
PE(1, 2) = \sin(0.01) \approx 0.0100
$$

$$
PE(1, 3) = \cos(0.01) \approx 0.9999
$$

So position 1 gets a unique fingerprint across all $d$ dimensions. Low-index dimensions swing fast; high-index dimensions swing slowly. Relative offsets between positions become linear functions of these encodings — that's why sinusoids can, in principle, extrapolate to sequence lengths never seen in training. Zero parameters to learn.

GPT-2 threw that out and **learned** $P$ directly — the table above. Tradeoffs I weighed building AL-1:

- **Learned:** flexible, often marginally better on in-distribution lengths, dead simple to implement (it's just another `Embedding`). Cost: hard-capped at $L = 1024$. Ask for position 1025 and there's no row — the model can't run.
- **Sinusoidal:** zero parameters, extrapolates past training length, but rigid and historically a touch worse on benchmarks.

For a toy GPT trained at fixed context length, learned wins on simplicity and I never need positions beyond $L$. I went learned. Modern models mostly moved past both to RoPE (rotates queries and keys by position inside attention rather than adding to the input) and ALiBi (distance-dependent score biases) — different mechanisms, same underlying problem.

---

## Where this fits in the build

AL-1 pipeline: [autograd](/blog/build-autograd-from-scratch/) → [gradient checker](/blog/numerical-gradient-checking-explained/) → [BPE tokenizer](/blog/byte-pair-encoding-from-scratch/) → [token embeddings](/blog/token-embeddings-explained/) → **positional encoding** → attention.

The input to the first transformer block is $X = E + P$ — token identity plus position, both learned, both lookups. Every token now carries two answers at once: *what am I* and *where am I*. Strip $P$ out and the model reads "dog bites man" exactly like "man bites dog," which is to say it reads nothing. That single 800k-parameter matrix, added at layer zero, is what lets the entire stack above it know order at all.

This is also where my two-table mental model finally locked in. Token embedding answers "what word is this?" Positional embedding answers "where does it sit?" Attention gets both in one vector and reasons over meaning *and* sequence simultaneously. Cheap to compute, easy to implement, and without it a transformer is a very expensive bag of words.