← Blogs

DeepSeek V4 Inside: One Token Through Every Block

How does DeepSeek V4 process one token? I follow the byte p in deepseek through CSA, HCA, mHC, MoE, and MTP on a toy width, with Flash-0731 sizes.

AI from Scratch Part 18 of 18

View as Markdown

In June I wrote a verification-first look at DeepSeek V4 claims because I could not find a paper, a config, or a model card that matched the video numbers. That post stays the checklist. This one is the other job: what actually happens inside the block when one byte goes through.

The paper is up. DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv 2606.19348.¹ The checkpoint I counted is DeepSeek-V4-Flash-0731. I am taking every width below from that config.json. The paper’s rounded Flash size is 284B total / 13B active; the Hub card for this file reports 304.2B. I am not reconciling those two here. Pro is 1.6T total / 49B active, same 1M context.

I cannot picture a 4096-d residual, 256 experts, or a million-token KV cache. So I locked a toy: the string deepseek as 8 raw bytes, and I painted the byte p (position 4, id 112) yellow in every figure. Hidden size 4, two residual streams, three layers so sliding-window attention, CSA, and HCA each show up once. Weights are one decimal. If a digit in a picture fights a table, the table wins. Image models smear numerals.

[!NOTE] Queries, keys, values, and the quadratic cost are in how attention works. The residual-stream arithmetic is in the transformer forward/backward pass. Byte lookup is token embeddings. Rotary positions are RoPE. The gated MLP is the SwiGLU section of the mini LLM walkthrough.

What does one Flash block look like?

Token ids, embedding, mHC expand into hc_mult = 4 residual streams, then 43 layers of attention plus DeepSeekMoE, each wrapped by mHC, then mix, RMSNorm, a main LM head for token $t+1$, and one MTP block for $t+2$.

flowchart TD
  ids["token ids"] --> emb["embedding"]
  emb --> exp["mHC expand, n_hc streams"]
  exp --> blk["block x 43"]
  blk --> mix["mHC mix + RMSNorm"]
  mix --> lm["LM head: token t+1"]
  mix --> mtp["MTP depth 1: token t+2"]
  subgraph one ["one block"]
    attn["SWA or CSA or HCA"]
    moe["DeepSeekMoE"]
    attn --> moe
  end
  blk --- one

compress_ratios in Flash-0731 starts [0, 0, 4, 128, ...]. Layers 0 and 1 are sliding-window only. After that, CSA ($m = 4$) and HCA ($m’ = 128$) alternate. The JSON lists a few trailing zeros after layer 42; num_hidden_layers is 43, so I am only reading the first 43 slots.

{
  "hidden_size": 4096,
  "num_hidden_layers": 43,
  "num_attention_heads": 64,
  "head_dim": 512,
  "num_key_value_heads": 1,
  "q_lora_rank": 1024,
  "qk_rope_head_dim": 64,
  "index_topk": 512,
  "sliding_window": 128,
  "n_routed_experts": 256,
  "num_experts_per_tok": 6,
  "n_shared_experts": 1,
  "moe_intermediate_size": 2048,
  "num_hash_layers": 3,
  "scoring_func": "sqrtsoftplus",
  "topk_method": "noaux_tc",
  "hc_mult": 4,
  "hc_sinkhorn_iters": 20,
  "num_nextn_predict_layers": 1,
  "vocab_size": 129280,
  "max_position_embeddings": 1048576,
  "swiglu_limit": 10.0
}

Knob	Toy (these figures)	Flash-0731	Pro (paper)
Total / active	n/a	284B / 13B	1.6T / 49B
Layers	3	43	n/a
Hidden $d$	4	4096	n/a
Heads $\times$ head dim	$2 \times 4$	$64 \times 512$	n/a
$q$ LoRA rank	n/a	1024	n/a
CSA $m$	2	4	n/a
HCA $m’$	4	128	n/a
Indexer top-$k$	2	512	n/a
Uncompressed window	2	128	n/a
RoPE dims	last 2	last 64	n/a
Routed / $k$ / shared	4 / 2 / 1	256 / 6 / 1	n/a
Expert intermediate	3	2048	n/a
Hash MoE layers	1 (layer 0)	3	n/a
$n_{hc}$ / Sinkhorn iters	2 / 3	4 / 20	n/a
MTP depth	1	1	1
Vocab	256 (bytes)	129280	n/a
Context	8	$1{,}048{,}576$	$1{,}000{,}000$

How does “deepseek” become an $8 \times 4$ matrix?

Raw bytes, not BPE. Ids [100, 101, 101, 112, 115, 101, 101, 107]. That is d e e p s e e k. Yellow p is ASCII 112.

I chose $E$ with one decimal. Yellow row:

$$ E_p = [0.6,\ -0.3,\ 0.4,\ 0.2] $$

Flash vocab is 129280. The toy vocab is 256 so the picture fits. Same gather: id $\to$ row of $E$. See token embeddings for the full-size version of that lookup.

How does mHC turn one stream into two?

Paper §2.2: the residual is $n_{hc}$ streams, not one vector of width $d$. Update

$$ X_{l+1} = B_l X_l + C_l F_l(A_l X_l) $$

$A$ mixes streams into the block. $C$ writes the block output back. $B$ mixes the residual and is projected onto doubly stochastic matrices by Sinkhorn-Knopp: $M^{(0)}=\exp(\tilde B)$, then alternate row-normalize and column-normalize, Flash hc_sinkhorn_iters = 20. I froze $A,B,C$ after this figure so the later arithmetic stays traceable. Flash would recompute a tiny input-dependent term each time ($\alpha = 0.01$ in the toy).

Expand with $w = [1.0,\ 0.4]$:

Stream	Yellow $p$
0	$[0.6,\ -0.3,\ 0.4,\ 0.2]$
1	$[0.24,\ -0.12,\ 0.16,\ 0.08]$

Mean-pool the whole $X$ into 8 numbers, RMSNorm, then the dynamic gates. After sigmoid / $2\cdot$sigmoid:

$$ A = [0.6002,\ 0.4504],\quad C = [1.0523,\ 0.8515] $$

$B$ after three Sinkhorn iterations (row sums and column sums both 1.0):

$$ B = \begin{bmatrix} 0.5741 & 0.4259 \\ 0.4259 & 0.5741 \end{bmatrix} $$

Stage	$M$	Row sums	Col sums
$\exp(B_{\text{raw}})$	$2.2381,\ 1.1101$ / $1.4957,\ 1.3482$	$3.3482,\ 2.8439$	$3.7338,\ 2.4583$
Iter 1 row	$0.6685,\ 0.3315$ / $0.5259,\ 0.4741$	$1.0,\ 1.0$	$1.1944,\ 0.8056$
Iter 1 col	$0.5597,\ 0.4115$ / $0.4403,\ 0.5885$	$0.9712,\ 1.0288$	$1.0,\ 1.0$
Iter 3 col $= B$	$0.5741,\ 0.4259$ / $0.4259,\ 0.5741$	$1.0,\ 1.0$	$1.0,\ 1.0$

A doubly stochastic $B$ cannot amplify the residual without bound as depth goes to 43. That is the point of the manifold constraint. The 20 iterations in Flash are cheap relative to attention at 1M tokens; they are not cheap relative to a 2×2 toy.

What does layer 0 sliding-window attention do to p?

Collapse streams: $H = A X$. Yellow:

$$ H_p = 0.6002\cdot[0.6,\ -0.3,\ 0.4,\ 0.2] + 0.4504\cdot[0.24,\ -0.12,\ 0.16,\ 0.08] = [0.4682,\ -0.2341,\ 0.3121,\ 0.1561] $$

Head 1, after $W_Q,W_K,W_V$:

	0	1	2	3
$Q$	0.1717	−0.0936	0.1951	0.1014
$K$	0.2185	−0.0702	0.0078	0.2887
$V$	0.1170	−0.0312	0.2185	0.0000

RoPE on the last 2 dims, position $i=3$: $\cos 3 = -0.9900$, $\sin 3 = 0.1411$.

$$ \begin{aligned} x' &= 0.1951(-0.9900) - 0.1014(0.1411) = -0.2074 \\ y' &= 0.1951(0.1411) + 0.1014(-0.9900) = -0.0729 \end{aligned} $$

Window 2, causal: $p$ sees the previous e and itself. Scale $1/\sqrt{4}=0.5$. Sink logit $s=-1.0$ goes in the denominator only, paper eq. (27):

$$ s_{h,i,j} = \frac{\exp(z_{h,i,j})}{\sum_k \exp(z_{h,i,k}) + \exp(z'_h)} $$

Key	$q\cdot k$	Score	$\exp$
pos 3 (e)	−0.0025	−0.0012	0.9988
pos 4 (p)	0.0749	0.0374	1.0382
sink	n/a	−1.0	0.3679

Denom $2.4048$. Weights $0.4153$ and $0.4317$. Sink mass $0.1530$. The keys do not sum to 1. That leftover is the sink: the head is allowed to attend to nothing.

$$ \text{ctx} = [0.0797,\ 0.0837,\ 0.0716,\ 0.0000] $$

Head 2 is a summary row in the figure. Concat both heads, grouped $W_O$ (Flash: 64 heads in o_groups = 8):

$$ F_{\text{attn}} = [0.0415,\ 0.1235,\ -0.0035,\ 0.0665] $$

Flash keeps a 128-token uncompressed window even after compression starts (sliding_window). The first two layers never compress. Local syntax is cheap. The 1M problem is the far tokens.

How does the residual update after attention?

$$ X_{\text{new}} = B X + C F $$

Yellow stream 0:

$$ \begin{aligned} BX_0 &= 0.5741\cdot[0.6,\ -0.3,\ 0.4,\ 0.2] + 0.4259\cdot[0.24,\ -0.12,\ 0.16,\ 0.08] \\ &= [0.4467,\ -0.2233,\ 0.2978,\ 0.1489] \\ C_0 F &= 1.0523\cdot F = [0.0437,\ 0.1300,\ -0.0037,\ 0.0699] \\ X_{\text{new},0} &= [0.4904,\ -0.0934,\ 0.2941,\ 0.2188] \end{aligned} $$

Stream 1 lands at $[0.4287,\ -0.0915,\ 0.2592,\ 0.1877]$.

Why does layer 0 MoE hash the token id?

Config: num_hash_layers = 3. Paper §2.1: the first MoE layers replace a learned router with hash routing, expert from the token id, no dense FFN at the start the way V3 had.

Toy: expert = token_id mod 4. Second routed expert is (e+1) mod 4 so top-2 still fires. Shared expert always on. Hash has no logits, so I fixed gates $0.7$ and $0.3$.

Token	Id	Primary	Second
d	100	0	1
e	101	1	2
e	101	1	2
p	112	0	1
s	115	3	0
e	101	1	2
e	101	1	2
k	107	3	0

Occupancy of the hashed primary: $[2,\ 4,\ 0,\ 2]$. Expert 2 is empty. That is why later layers switch to a learned router plus noaux_tc bias.

Input to the expert, after mixing the post-attention streams:

$$ h_p = [0.4874,\ -0.0973,\ 0.2933,\ 0.2159] $$

Expert 0 is SwiGLU, intermediate 3 (Flash 2048), output clamped to $[-10,10]$ (swiglu_limit).

$$ \text{gate} = h W_{\text{gate}} = [0.2109,\ 0.0449,\ 0.0533] $$

$\mathrm{SiLU}(g)=g\cdot\sigma(g)$:

$g$	$\sigma(g)$	SiLU	$\times$ up	hid
0.2109	0.5525	0.1166	0.1520	0.0177
0.0449	0.5112	0.0230	−0.0229	−0.0005
0.0533	0.5133	0.0274	0.2852	0.0078

$y = \text{hid}\,W_{\text{down}} = [0.0062,\ 0.0032,\ 0.0013,\ 0.0044]$. Clamp does not bite. On FP8/FP4 training it does; that is why the limit exists (paper §4.2.3).

Combine: shared $+$ gated routed.

Path	Vector
Shared	$[0.0030,\ 0.0037,\ 0.0074,\ 0.0043]$
Expert 0	$[0.0062,\ 0.0032,\ 0.0013,\ 0.0044]$
Expert 1	$[0.0068,\ 0.0029,\ -0.0004,\ 0.0039]$
$0.7 e_0 + 0.3 e_1$	$[0.0064,\ 0.0031,\ 0.0008,\ 0.0043]$
$F_{\text{MoE}}$	$[0.0094,\ 0.0068,\ 0.0082,\ 0.0085]$

Second mHC wrap. $X$ after layer 0, yellow:

Stream	$X$
0	$[0.4739,\ -0.0854,\ 0.2879,\ 0.2146]$
1	$[0.4629,\ -0.0865,\ 0.2811,\ 0.2082]$

Flash activates 6 of 256 routed experts plus 1 shared. That is the 13B-active number sitting inside 284B total.

How does CSA compress 8 tokens into 4 KV entries?

Paper §2.3.1: every $m$ tokens become one KV entry by an overlapped weighted sum of two streams $C^a$ and $C^b$. Adjacent blocks share tokens. Then a lightning indexer keeps top-$k$ (Flash 512). Core attention is shared-KV MQA: the compressed entry is both $K$ and $V$. An uncompressed window of sliding_window tokens is concatenated so a query can see inside its own compression block without breaking causality.

Toy $m=2$, overlap weights $C^a=[0.5,\ 0.3]$, $C^b=[0.2]$ on the next token. Entry 2 uses tokens of entry 1. $H$ after mixing layer-0 $X$, yellow $[0.4930,\ -0.0902,\ 0.2994,\ 0.2226]$.

Entry	Token indices	Compressed KV
0	0, 1, 2	$[0.3717,\ 0.3422,\ -0.0321,\ 0.0540]$
1	2, 3, 4	$[0.2972,\ 0.2917,\ 0.1721,\ 0.0681]$
2	4, 5, 6	$[0.1243,\ 0.4243,\ 0.1979,\ 0.1469]$
3	6, 7	$[0.1889,\ 0.1770,\ 0.1481,\ -0.0288]$

Entry 1 contains future token s (position 5) relative to query $p$. Causal rule: an entry is visible only if every token in it has position $\le$ the query. Entry 1 is forbidden. The indexer can still score it. Core attention must not.

[!GOTCHA] Flash CSA softmax-normalizes $2m$ overlap weights per block and uses learned $C^a$, $C^b$ plus positional biases. The toy keeps three weights that sum to 1 so the grid stays countable. Same idea, not the same tensor.

RMSNorm on the Q head and on the compressed KV head, then partial RoPE on the last 2 dims (Flash last 64). Indexer scores of $p$ against the four entries, ReLU, rank:

Entry	Score	ReLU	Causal?
0	0.0891	0.0891	yes
1	0.1076	0.1076	no (contains s)
2	0.0923	0.0923	no
3	0.0775	0.0775	no

Top-2 by score would be 1 then 2. After the mask, only entry 0 remains. Window $n_{\text{win}}=2$ adds uncompressed tokens 3 and 4 (e and p). Keys for core attention: 1 compressed + 2 window.

Scores for $p$, sink in the denom: $[0.9980,\ 0.1329,\ 1.9828]$, weights $[0.2362,\ 0.0994,\ 0.6323]$, sink mass $0.032$. Context $[1.4511,\ 0.2906,\ -0.6491,\ -0.3722]$.

Then RoPE with position $-i$ on the last 2 dims of that output. Paper: the compressed KV already carries absolute positions, so the weighted sum would leak them unless you rotate the output by $-i$. After that, grouped $W_O$:

$$ F_{\text{CSA}} = [0.8717,\ 0.9787,\ 0.0663,\ 0.6627] $$

Learned router on the post-CSA hidden. Affinity is $\sqrt{\mathrm{softplus}(x)}$, not V3’s sigmoid (scoring_func": "sqrtsoftplus"). Bias $b=[0.0,\ 0.1,\ -0.1,\ 0.0]$ is added only for top-k selection (topk_method": "noaux_tc"). Gates are renormalized from the unbiased scores.

Expert	Logit	softplus	$\sqrt{\cdot}$	$+$ bias	Selected
0	0.6466	1.0679	1.0334	1.0334	yes
1	0.0176	0.7020	0.8378	0.9378	no
2	0.3729	0.8969	0.9470	0.8470	no
3	1.0715	1.3661	1.1688	1.1688	yes

Top-2: experts 3 and 0. Gates $0.5307$ and $0.4693$.

What does HCA change?

Same MQA, window, RoPE, sink. Compression $m’=4$ (Flash 128), no overlap, no indexer. Dense attention over $n/m’$ entries. At toy $T=8$ that is 2 entries: tokens 0-3 and 4-7. Yellow $p$ lives in entry 0. Entry 1 still contains the future, so a causal query at $p$ only sees entry 0 plus the 2-token window.

KV entries per query at Flash context $1{,}048{,}576$:

Attention	Entries the query scores	How
SWA	128	sliding_window
CSA	$512 + 128 = 640$	indexer top-$k$ + window
HCA	$8192 + 128 = 8320$	$1{,}048{,}576/128$ dense compressed + window
Plain dense	$1{,}048{,}576$	every past token

CSA is the sparse one. HCA is the heavy shrink with no indexer. Interleaving them is the hybrid in compress_ratios. Paper §2.3.4: mixed BF16 on the RoPE dims and FP8 on the rest, FP4 in the indexer, and versus a BF16 GQA-8 head-dim-128 baseline the 1M KV cache is about 2%. Versus DeepSeek-V3.2, Pro at 1M is 27% of the single-token FLOPs and 10% of the KV; Flash is 10% of the FLOPs and 7% of the KV.

How does noaux_tc move expert bias?

No auxiliary load-balancing loss. After a batch, if expert $i$ ran hot, nudge $b_i$ down so the next batch is less likely to pick it. If it ran cold, nudge $b_i$ up. The bias does not enter the gate values, only the top-k pick. A small sequence-wise balance term stops one sequence from dumping everything on one expert (paper §2.1). Hash layers do not learn this; layers 0-2 in Flash skip the router.

On the toy hash layer, occupancy $[2,4,0,2]$ already shows the failure mode a learned bias is meant to fix: one expert empty, one stuffed with every e.

What does the main head predict for t+1?

Final mix of the two streams into $[8\times 4]$, RMSNorm, LM head. Vocab 256 in the toy; I only score four candidate bytes: e (101), k (107), space (32), s (115). True next byte after $p$ is s.

Where does the extra t+2 head get its input?

MTP depth 1, same wiring as V3 (paper §2.1: “without modification”). During training, concatenate the main hidden state of $p$ with the embedding of the true next byte s, RMSNorm, a small projection, one transformer block, a second head. Inference discards this path.

What is the loss, and who updates which weights?

V3 used $L = L_{\text{lm}} + 0.3\, L_{\text{mtp}}$. V4 keeps that MTP setup. $L_{\text{lm}}$ is cross entropy on $t+1$ (s for this $p$). $L_{\text{mtp}}$ is cross entropy on $t+2$ (e). See cross entropy if the softmax-NLL step is rusty.

Muon (paper Algorithm 1, §2.4) updates the 2D weight matrices: Newton-Schulz orthogonalization of the momentum, then a scaled step. Embeddings, RMSNorm gains, and mHC static gates stay on AdamW. Training is FP8/FP4 on clusters, $32$T+ tokens, then YaRN from 64K original_max_position_embeddings to $1{,}048{,}576$ (rope_scaling.type = yarn, factor 16).

What is stored at generate time?

Main head only. The KV cache is compressed entries, not per-token $K,V$ for every layer. After 8 toy tokens: 4 CSA entries, 2 HCA entries. The next byte grows the cache by a fraction of an entry ($1/m$ or $1/m’$), not by a full head-dim vector per layer the way a vanilla transformer does. Flash also keeps the 128-token uncompressed window.

That is the 1M trick in one picture: you are not storing a million keys per head. You are storing $n/4$ or $n/128$ mixed slots, plus 128 locals, plus (for CSA) a top-512 gather.

Can I see p’s numbers on one page?

Five equations I actually used:

$$ \begin{aligned} X_{l+1} &= B_l X_l + C_l F_l(A_l X_l) \\ \text{affinity} &= \sqrt{\mathrm{softplus}(x)} \\ C^{\text{Comp}}_i &= \text{overlapped weighted sum of block }i \\ s_{ij} &= \frac{\exp(z_{ij})}{\sum_k\exp(z_{ik})+\exp(z')} \\ L &= L_{\text{lm}} + \lambda L_{\text{mtp}} \end{aligned} $$

Eight Flash numbers worth keeping: 43 layers, $d=4096$, $m=4$, $m’=128$, indexer 512, window 128, 256 routed / top-6 / 1 shared, context $1{,}048{,}576$.

Shapes I would check first if I implemented the toy (batch 1, T = 8)

1. Token ids [8], embedding [8, 4].
2. Residual $X$ [2, 8, 4].
3. $A X$ [8, 4].
4. Per-head $Q$ [8, 4]; CSA/HCA compressed KV [T/m, 4] or [T/m', 4].
5. SWA scores [8, window] with window 2.
6. CSA indexer scores [8, 4] then a top-2 index tensor.
7. MoE gates [8, 2] plus a shared output [8, 4].
8. Expert hidden [8, 3] inside SwiGLU.
9. LM logits toy [8, 256]; MTP hidden concat [8, 8] before the projection back to 4.
10. Cache after 8 tokens: CSA length 4, HCA length 2, window 2.

Read next: the June V4 engineering notes for the verification checklist and the $n^2$ cost, then how attention works if the score-row still feels like a vibe. The paper HTML is arXiv 2606.19348. The file I counted is Flash-0731 config.json.

1. DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, arXiv 2606.19348. Citation date 26 April 2026. Hybrid CSA/HCA §2.3, mHC §2.2, MoE and hash layers §2.1, MTP same as V3 §2.1, Muon §2.4, Flash/Pro widths in the abstract and §4.2.1. Checkpoints: deepseek-ai/deepseek-v4. ↩

Support the writing

If this post helped, a coffee keeps the deep dives coming.

Buy me a coffee

Related posts

• DeepSeek V4 Explained: Long-Context Engineering and Math Jun 5, 2026 · 14 min
• Looped Transformers Explained: Recurrent Depth and Astra Sep 3, 2026 · 14 min
• Build a Mini LLM from Scratch in NumPy: RoPE, GQA, SwiGLU Jul 16, 2026 · 11 min