Positional Encoding Explained: How Transformers Learn Word Order

I closed the token embeddings post with a problem I’d waved away: the embedding lookup is order-blind. gather returns the same vector for cat whether it’s the first word or the hundredth. And attention — the thing those vectors flow into — is permutation-invariant. Shuffle the tokens and, without help, the math produces the same set of outputs in shuffled order. A transformer is, by default, a bag-of-words machine. That broke my mental model the first time I really sat with it, because order is most of language.

$$ \text{"Dog bites man"} \neq \text{"Man bites dog"}. $$

Same three words. Same three token embeddings. Opposite meaning. Something has to tell the model where each token sits. That something is positional encoding. In GPT it’s a second lookup table living right next to the token embedding matrix. I’ll walk through why attention can’t see order on its own, what positional vectors actually are, the math of adding them, a full three-word worked example with every addition written out, learned vs sinusoidal encodings with formula steps, and where this lands in my AL-1 build.

Self-attention computes a weighted sum over all tokens. The weights come from dot products between query and key vectors. A dot product is just

$$ q_i \cdot k_j = \sum_{k=1}^{d} q_{i,k}\, k_{j,k}. $$

There is no position index $t$ inside that sum. The number $q_i \cdot k_j$ depends only on the vectors, not on whether $i$ is slot 0 or slot 99. Shuffle the tokens and you shuffle the same vectors into different slots — attention still computes the same pairwise dot products, just permuted. Without extra information, the model genuinely cannot tell position 0 from position 100.

Picture six people in a room — Alice, Bob, Charlie, David, Eva, Frank. You’re told who is present but not where they’re sitting. That’s a transformer with token embeddings alone: it knows the cast, not the seating chart. Language needs the seating chart.

flowchart LR
  subgraph blind["Token embeddings only"]
    a1["cat @ pos 0"] --> v1["vector V"]
    a2["cat @ pos 99"] --> v1
  end
  subgraph fixed["+ Positional encoding"]
    b1["cat @ pos 0"] --> w1["V + P0"]
    b2["cat @ pos 99"] --> w2["V + P99"]
  end

Same word cat, same embedding vector $V$. Only after adding a position-specific vector $P_t$ does slot 0 become $V + P_0$ and slot 99 become $V + P_{99}$ — two different inputs to attention.

What is a positional embedding?

The intuition is a second lookup table, parallel to the token table. Token embedding answers “what word is this?” Positional embedding answers “where does it sit?”

Stack all position vectors into a matrix

$$ P \in \mathbb{R}^{L \times d} $$

where $L$ is the maximum sequence length and $d$ is d_model. Row $t$ is the vector that means “I am the token at position $t$.”

GPT-2 small uses

$$ L = 1024 $$

$$ d = 768 $$

$$ P \in \mathbb{R}^{1024 \times 768} $$

$$ 1024 \times 768 = 786{,}432 \text{ parameters} $$

Roughly 800k parameters whose only job is to encode order. It’s the same gather lookup as token embeddings, indexed by position ID instead of token ID.

Toy table with $L = 5$, $d = 2$:

$$ P = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \\ 0.5 & 0.6 \\ 0.7 & 0.8 \\ 0.9 & 1.0 \end{bmatrix} $$

$$ \text{row } 0 = \text{first token} $$

$$ \text{row } 1 = \text{second token} $$

$$ \vdots $$

The implementation is two lines longer than the embedding layer, because the indices are generated, not passed in:

class PositionalEmbedding:
    def __init__(self, max_seq, d_model, std=0.02):
        w = np.random.randn(max_seq, d_model) * std
        self.weight = Tensor(w)              # (max_seq, d_model), trainable

    def forward(self, seq_len):
        positions = np.arange(seq_len)       # [0, 1, 2, ..., seq_len-1]
        return self.weight.gather(positions, dim=0)   # (seq_len, d_model)

For a 5-token sentence, np.arange(5) makes [0, 1, 2, 3, 4] and gather pulls those five rows. Same $0.02$ GPT-2 initializer as the token embeddings — start small so training isn’t fighting huge activations on step one. Cost is $O(L)$ row selection, no matmul. Next to attention’s $O(L^2)$, positional lookup is rounding error.

The math: why we add, not concatenate

The transformer input is the sum of the two tables, elementwise:

$$ X = E + P $$

where $E$ holds token embeddings and $P$ holds position embeddings, both with shape $(L, d)$.

Take a toy sentence of three tokens with $d = 2$:

$$ E = \begin{bmatrix} 1 & 2 \\ 3 & 4 \\ 5 & 6 \end{bmatrix} $$

$$ P = \begin{bmatrix} 0.1 & 0.2 \\ 0.3 & 0.4 \\ 0.5 & 0.6 \end{bmatrix} $$

Row 0 of $X$:

$$ X_{0,0} = E_{0,0} + P_{0,0} $$

$$ X_{0,0} = 1 + 0.1 $$

$$ X_{0,0} = 1.1 $$

$$ X_{0,1} = E_{0,1} + P_{0,1} $$

$$ X_{0,1} = 2 + 0.2 $$

$$ X_{0,1} = 2.2 $$

Row 1 of $X$:

$$ X_{1,0} = 3 + 0.3 = 3.3 $$

$$ X_{1,1} = 4 + 0.4 = 4.4 $$

Row 2 of $X$:

$$ X_{2,0} = 5 + 0.5 = 5.5 $$

$$ X_{2,1} = 6 + 0.6 = 6.6 $$

$$ X = E + P = \begin{bmatrix} 1.1 & 2.2 \\ 3.3 & 4.4 \\ 5.5 & 6.6 \end{bmatrix}. $$

That tiny addition is the whole trick. After it, cat at position 0 and cat at position 99 are genuinely different vectors going into attention.

Why add two unrelated things instead of concatenating? Concatenation would stick $[1, 2]$ and $[0.1, 0.2]$ into $[1, 2, 0.1, 0.2]$, doubling $d$. Attention’s projection cost scales with $d^2$, so doubling width quadruples that cost and inflates every downstream weight. Addition keeps $d$ fixed — free position information, zero width tax. The model has 768 dimensions to play with; gradient descent learns to carve out subspaces where token identity and position don’t collide. The sum stays disentangleable enough for attention to read both.

Like token embeddings, $P$ starts as scaled noise. Backprop only updates the positions a batch actually used — a 5-token sequence sends gradients into rows 0–4 and leaves rows 5–1023 alone that step. Over training the rows specialize. Nobody hand-codes any of it; it falls out of next-token prediction, same as everything else.

Worked example: “dog bites man” vs “man bites dog”

I’ll use $d = 2$ so every addition fits on one line. Three token embeddings from the token embedding table:

$$ e_{\text{dog}} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} $$

$$ e_{\text{bites}} = \begin{bmatrix} 3 \\ 1 \end{bmatrix} $$

$$ e_{\text{man}} = \begin{bmatrix} 0 \\ 3 \end{bmatrix} $$

Three learned position rows:

$$ p_0 = \begin{bmatrix} 0.1 \\ 0.0 \end{bmatrix} $$

$$ p_1 = \begin{bmatrix} 0.0 \\ 0.2 \end{bmatrix} $$

$$ p_2 = \begin{bmatrix} 0.3 \\ 0.1 \end{bmatrix} $$

Sentence 1: “dog bites man”

Token at position 0 is dog. Combine embedding with position 0:

$$ x_0 = e_{\text{dog}} + p_0 $$

$$ x_{0,0} = 1 + 0.1 $$

$$ x_{0,0} = 1.1 $$

$$ x_{0,1} = 2 + 0.0 $$

$$ x_{0,1} = 2.0 $$

$$ x_0 = \begin{bmatrix} 1.1 \\ 2.0 \end{bmatrix} $$

Token at position 1 is bites:

$$ x_1 = e_{\text{bites}} + p_1 $$

$$ x_{1,0} = 3 + 0.0 $$

$$ x_{1,0} = 3.0 $$

$$ x_{1,1} = 1 + 0.2 $$

$$ x_{1,1} = 1.2 $$

$$ x_1 = \begin{bmatrix} 3.0 \\ 1.2 \end{bmatrix} $$

Token at position 2 is man:

$$ x_2 = e_{\text{man}} + p_2 $$

$$ x_{2,0} = 0 + 0.3 $$

$$ x_{2,0} = 0.3 $$

$$ x_{2,1} = 3 + 0.1 $$

$$ x_{2,1} = 3.1 $$

$$ x_2 = \begin{bmatrix} 0.3 \\ 3.1 \end{bmatrix} $$

Full input matrix for “dog bites man”:

$$ X_{\text{dbm}} = \begin{bmatrix} 1.1 & 2.0 \\ 3.0 & 1.2 \\ 0.3 & 3.1 \end{bmatrix} $$

Sentence 2: “man bites dog”

Same three token embeddings. Same three position rows. Different order — so each token meets a different position vector.

Token at position 0 is man:

$$ x'_0 = e_{\text{man}} + p_0 $$

$$ x'_{0,0} = 0 + 0.1 $$

$$ x'_{0,0} = 0.1 $$

$$ x'_{0,1} = 3 + 0.0 $$

$$ x'_{0,1} = 3.0 $$

$$ x'_0 = \begin{bmatrix} 0.1 \\ 3.0 \end{bmatrix} $$

Token at position 1 is bites:

$$ x'_1 = e_{\text{bites}} + p_1 $$

$$ x'_{1,0} = 3 + 0.0 $$

$$ x'_{1,0} = 3.0 $$

$$ x'_{1,1} = 1 + 0.2 $$

$$ x'_{1,1} = 1.2 $$

$$ x'_1 = \begin{bmatrix} 3.0 \\ 1.2 \end{bmatrix} $$

Token at position 2 is dog:

$$ x'_2 = e_{\text{dog}} + p_2 $$

$$ x'_{2,0} = 1 + 0.3 $$

$$ x'_{2,0} = 1.3 $$

$$ x'_{2,1} = 2 + 0.1 $$

$$ x'_{2,1} = 2.1 $$

$$ x'_2 = \begin{bmatrix} 1.3 \\ 2.1 \end{bmatrix} $$

Full input matrix for “man bites dog”:

$$ X_{\text{mbd}} = \begin{bmatrix} 0.1 & 3.0 \\ 3.0 & 1.2 \\ 1.3 & 2.1 \end{bmatrix} $$

What changed?

Compare row by row. Position 0 used to be dog at $[1.1, 2.0]$; now it’s man at $[0.1, 3.0]$. Position 2 used to be man at $[0.3, 3.1]$; now it’s dog at $[1.3, 2.1]$. Only position 1 (bites) lands on the same combined vector $[3.0, 1.2]$ — because bites stayed in the middle.

Without positional encoding, sentence 1 would be $[e_{\text{dog}}, e_{\text{bites}}, e_{\text{man}}]$ and sentence 2 would be $[e_{\text{man}}, e_{\text{bites}}, e_{\text{dog}}]$ — a permutation of the same three vectors. Attention is blind to that shuffle. With $P$ added, $X_{\text{dbm}} \neq X_{\text{mbd}}$ as matrices. Query-key dot products inside attention now produce different scores. The model can finally tell who did the biting.

Learned vs sinusoidal: the design fork

The original Attention Is All You Need paper used fixed sinusoidal encodings — no learned parameters, computed from a formula. For position $t$ and dimension index $i$ (with model width $d$):

$$ PE(t, 2i) = \sin\!\left(\frac{t}{10000^{2i/d}}\right) $$

$$ PE(t, 2i+1) = \cos\!\left(\frac{t}{10000^{2i/d}}\right) $$

Even dimensions get sines. Odd dimensions get cosines. Each dimension oscillates at a different wavelength because the divisor $10000^{2i/d}$ grows with $i$.

Worked numbers for $t = 1$, $i = 0$, $d = 4$:

$$ \frac{2i}{d} = \frac{0}{4} = 0 $$

$$ 10000^{0} = 1 $$

$$ \frac{t}{10000^{2i/d}} = \frac{1}{1} = 1 $$

$$ PE(1, 0) = \sin(1) \approx 0.8415 $$

$$ PE(1, 1) = \cos(1) \approx 0.5403 $$

For $t = 1$, $i = 1$:

$$ \frac{2i}{d} = \frac{2}{4} = 0.5 $$

$$ 10000^{0.5} = 100 $$

$$ \frac{t}{10000^{2i/d}} = \frac{1}{100} = 0.01 $$

$$ PE(1, 2) = \sin(0.01) \approx 0.0100 $$

$$ PE(1, 3) = \cos(0.01) \approx 0.9999 $$

So position 1 gets a unique fingerprint across all $d$ dimensions. Low-index dimensions swing fast; high-index dimensions swing slowly. Relative offsets between positions become linear functions of these encodings — that’s why sinusoids can, in principle, extrapolate to sequence lengths never seen in training. Zero parameters to learn.

GPT-2 threw that out and learned $P$ directly — the table above. Tradeoffs I weighed building AL-1:

Learned: flexible, often marginally better on in-distribution lengths, dead simple to implement (it’s just another Embedding). Cost: hard-capped at $L = 1024$. Ask for position 1025 and there’s no row — the model can’t run.
Sinusoidal: zero parameters, extrapolates past training length, but rigid and historically a touch worse on benchmarks.

For a toy GPT trained at fixed context length, learned wins on simplicity and I never need positions beyond $L$. I went learned. Modern models mostly moved past both to RoPE (rotates queries and keys by position inside attention rather than adding to the input) and ALiBi (distance-dependent score biases) — different mechanisms, same underlying problem.

Where this fits in the build

AL-1 pipeline: autograd → gradient checker → BPE tokenizer → token embeddings → positional encoding → attention.

The input to the first transformer block is $X = E + P$ — token identity plus position, both learned, both lookups. Every token now carries two answers at once: what am I and where am I. Strip $P$ out and the model reads “dog bites man” exactly like “man bites dog,” which is to say it reads nothing. That single 800k-parameter matrix, added at layer zero, is what lets the entire stack above it know order at all.

This is also where my two-table mental model finally locked in. Token embedding answers “what word is this?” Positional embedding answers “where does it sit?” Attention gets both in one vector and reasons over meaning and sequence simultaneously. Cheap to compute, easy to implement, and without it a transformer is a very expensive bag of words.