---
layout: post
title: "DeepSeek V4 Inside: One Token Through Every Block"
date: 2026-09-04 17:00:00 +0530
tags: ["DeepSeek", "LLM", "Attention", "Long Context", "Sparse Attention", "Transformers", "RoPE", "SwiGLU", "Model Training"]
category: "ai"
description: "How does DeepSeek V4 process one token? I follow the byte p in deepseek through CSA, HCA, mHC, MoE, and MTP on a toy width, with Flash-0731 sizes."
keywords: "deepseek v4 inside, how deepseek v4 processes a token, CSA compressed sparse attention, HCA heavily compressed attention, mHC sinkhorn, deepseek moe sqrtsoftplus, flash-0731 config, multi-token prediction"
image: /assets/images/og-deepseek-v4-inside-one-token.png
order: 18
---

In June I wrote a [verification-first look at DeepSeek V4 claims](/blog/deepseek-v4-engineering-explained/) because I could not find a paper, a config, or a model card that matched the video numbers. That post stays the checklist. This one is the other job: what actually happens inside the block when one byte goes through.

The paper is up. DeepSeek-AI, *DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence*, [arXiv 2606.19348](https://arxiv.org/abs/2606.19348).[^paper] The checkpoint I counted is [DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731). I am taking every width below from that `config.json`. The paper's rounded Flash size is 284B total / 13B active; the Hub card for this file reports 304.2B. I am not reconciling those two here. Pro is 1.6T total / 49B active, same 1M context.

I cannot picture a 4096-d residual, 256 experts, or a million-token KV cache. So I locked a toy: the string `deepseek` as 8 raw bytes, and I painted the byte **p** (position 4, id 112) yellow in every figure. Hidden size 4, two residual streams, three layers so sliding-window attention, CSA, and HCA each show up once. Weights are one decimal. If a digit in a picture fights a table, the table wins. Image models smear numerals.

> [!NOTE]
> Queries, keys, values, and the quadratic cost are in [how attention works](/blog/attention-in-transformers-explained/). The residual-stream arithmetic is in the [transformer forward/backward pass](/blog/transformer-from-scratch-forward-backward-math/). Byte lookup is [token embeddings](/blog/token-embeddings-explained/). Rotary positions are [RoPE](/blog/positional-encoding-explained/). The gated MLP is the [SwiGLU section of the mini LLM walkthrough](/blog/build-mini-llm-numpy-from-scratch/).

## What does one Flash block look like?

Token ids, embedding, mHC expand into `hc_mult = 4` residual streams, then 43 layers of attention plus DeepSeekMoE, each wrapped by mHC, then mix, RMSNorm, a main LM head for token $t+1$, and one MTP block for $t+2$.

```mermaid
flowchart TD
  ids["token ids"] --> emb["embedding"]
  emb --> exp["mHC expand, n_hc streams"]
  exp --> blk["block x 43"]
  blk --> mix["mHC mix + RMSNorm"]
  mix --> lm["LM head: token t+1"]
  mix --> mtp["MTP depth 1: token t+2"]
  subgraph one ["one block"]
    attn["SWA or CSA or HCA"]
    moe["DeepSeekMoE"]
    attn --> moe
  end
  blk --- one
```

`compress_ratios` in Flash-0731 starts `[0, 0, 4, 128, ...]`. Layers 0 and 1 are sliding-window only. After that, CSA ($m = 4$) and HCA ($m' = 128$) alternate. The JSON lists a few trailing zeros after layer 42; `num_hidden_layers` is 43, so I am only reading the first 43 slots.

```json
{
  "hidden_size": 4096,
  "num_hidden_layers": 43,
  "num_attention_heads": 64,
  "head_dim": 512,
  "num_key_value_heads": 1,
  "q_lora_rank": 1024,
  "qk_rope_head_dim": 64,
  "index_topk": 512,
  "sliding_window": 128,
  "n_routed_experts": 256,
  "num_experts_per_tok": 6,
  "n_shared_experts": 1,
  "moe_intermediate_size": 2048,
  "num_hash_layers": 3,
  "scoring_func": "sqrtsoftplus",
  "topk_method": "noaux_tc",
  "hc_mult": 4,
  "hc_sinkhorn_iters": 20,
  "num_nextn_predict_layers": 1,
  "vocab_size": 129280,
  "max_position_embeddings": 1048576,
  "swiglu_limit": 10.0
}
```
{: data-title="DeepSeek-V4-Flash-0731/config.json"}

| Knob | Toy (these figures) | Flash-0731 | Pro (paper) |
|---|---:|---:|---:|
| Total / active | n/a | 284B / 13B | 1.6T / 49B |
| Layers | 3 | 43 | n/a |
| Hidden $d$ | 4 | 4096 | n/a |
| Heads $\times$ head dim | $2 \times 4$ | $64 \times 512$ | n/a |
| $q$ LoRA rank | n/a | 1024 | n/a |
| CSA $m$ | 2 | 4 | n/a |
| HCA $m'$ | 4 | 128 | n/a |
| Indexer top-$k$ | 2 | 512 | n/a |
| Uncompressed window | 2 | 128 | n/a |
| RoPE dims | last 2 | last 64 | n/a |
| Routed / $k$ / shared | 4 / 2 / 1 | 256 / 6 / 1 | n/a |
| Expert intermediate | 3 | 2048 | n/a |
| Hash MoE layers | 1 (layer 0) | 3 | n/a |
| $n_{hc}$ / Sinkhorn iters | 2 / 3 | 4 / 20 | n/a |
| MTP depth | 1 | 1 | 1 |
| Vocab | 256 (bytes) | 129280 | n/a |
| Context | 8 | $1{,}048{,}576$ | $1{,}000{,}000$ |

![Full map of DeepSeek V4: bytes to embedding to mHC streams to 43 hybrid-attention MoE blocks to LM head and MTP](/assets/images/deepseek-v4-inside/fig01-full-map.webp "Figure 1: full map. Yellow is p. Flash strip is SWA, SWA, then 4/128 alternating.")

## How does "deepseek" become an $8 \times 4$ matrix?

Raw bytes, not BPE. Ids `[100, 101, 101, 112, 115, 101, 101, 107]`. That is `d e e p s e e k`. Yellow **p** is ASCII 112.

![Byte tokenisation of deepseek and the 8 by 4 embedding matrix with row p highlighted](/assets/images/deepseek-v4-inside/fig02-byte-embedding.webp "Figure 2: embedding E, shape [8 x 4].")

I chose $E$ with one decimal. Yellow row:

$$
E_p = [0.6,\ -0.3,\ 0.4,\ 0.2]
$$

Flash vocab is 129280. The toy vocab is 256 so the picture fits. Same gather: id $\to$ row of $E$. See [token embeddings](/blog/token-embeddings-explained/) for the full-size version of that lookup.

## How does mHC turn one stream into two?

Paper §2.2: the residual is $n_{hc}$ streams, not one vector of width $d$. Update

$$
X_{l+1} = B_l X_l + C_l F_l(A_l X_l)
$$

$A$ mixes streams into the block. $C$ writes the block output back. $B$ mixes the residual and is projected onto doubly stochastic matrices by Sinkhorn-Knopp: $M^{(0)}=\exp(\tilde B)$, then alternate row-normalize and column-normalize, Flash `hc_sinkhorn_iters = 20`. I froze $A,B,C$ after this figure so the later arithmetic stays traceable. Flash would recompute a tiny input-dependent term each time ($\alpha = 0.01$ in the toy).

Expand with $w = [1.0,\ 0.4]$:

| Stream | Yellow $p$ |
|---|---|
| 0 | $[0.6,\ -0.3,\ 0.4,\ 0.2]$ |
| 1 | $[0.24,\ -0.12,\ 0.16,\ 0.08]$ |

Mean-pool the whole $X$ into 8 numbers, RMSNorm, then the dynamic gates. After sigmoid / $2\cdot$sigmoid:

$$
A = [0.6002,\ 0.4504],\quad C = [1.0523,\ 0.8515]
$$

$B$ after three Sinkhorn iterations (row sums and column sums both 1.0):

$$
B = \begin{bmatrix} 0.5741 & 0.4259 \\ 0.4259 & 0.5741 \end{bmatrix}
$$

| Stage | $M$ | Row sums | Col sums |
|---|---|---|---|
| $\exp(B_{\text{raw}})$ | $2.2381,\ 1.1101$ / $1.4957,\ 1.3482$ | $3.3482,\ 2.8439$ | $3.7338,\ 2.4583$ |
| Iter 1 row | $0.6685,\ 0.3315$ / $0.5259,\ 0.4741$ | $1.0,\ 1.0$ | $1.1944,\ 0.8056$ |
| Iter 1 col | $0.5597,\ 0.4115$ / $0.4403,\ 0.5885$ | $0.9712,\ 1.0288$ | $1.0,\ 1.0$ |
| Iter 3 col $= B$ | $0.5741,\ 0.4259$ / $0.4259,\ 0.5741$ | $1.0,\ 1.0$ | $1.0,\ 1.0$ |

![mHC expansion of E into two residual streams and Sinkhorn iterations on mixer B](/assets/images/deepseek-v4-inside/fig03-mhc-expand.webp "Figure 3: mHC expand. B ends doubly stochastic.")

A doubly stochastic $B$ cannot amplify the residual without bound as depth goes to 43. That is the point of the manifold constraint. The 20 iterations in Flash are cheap relative to attention at 1M tokens; they are not cheap relative to a 2×2 toy.

## What does layer 0 sliding-window attention do to p?

Collapse streams: $H = A X$. Yellow:

$$
H_p = 0.6002\cdot[0.6,\ -0.3,\ 0.4,\ 0.2] + 0.4504\cdot[0.24,\ -0.12,\ 0.16,\ 0.08] = [0.4682,\ -0.2341,\ 0.3121,\ 0.1561]
$$

Head 1, after $W_Q,W_K,W_V$:

| | 0 | 1 | 2 | 3 |
|---|---:|---:|---:|---:|
| $Q$ | 0.1717 | −0.0936 | 0.1951 | 0.1014 |
| $K$ | 0.2185 | −0.0702 | 0.0078 | 0.2887 |
| $V$ | 0.1170 | −0.0312 | 0.2185 | 0.0000 |

RoPE on the last 2 dims, position $i=3$: $\cos 3 = -0.9900$, $\sin 3 = 0.1411$.

$$
\begin{aligned}
x' &= 0.1951(-0.9900) - 0.1014(0.1411) = -0.2074 \\
y' &= 0.1951(0.1411) + 0.1014(-0.9900) = -0.0729
\end{aligned}
$$

Window 2, causal: $p$ sees the previous `e` and itself. Scale $1/\sqrt{4}=0.5$. Sink logit $s=-1.0$ goes in the **denominator** only, paper eq. (27):

$$
s_{h,i,j} = \frac{\exp(z_{h,i,j})}{\sum_k \exp(z_{h,i,k}) + \exp(z'_h)}
$$

| Key | $q\cdot k$ | Score | $\exp$ |
|---|---:|---:|---:|
| pos 3 (`e`) | −0.0025 | −0.0012 | 0.9988 |
| pos 4 (`p`) | 0.0749 | 0.0374 | 1.0382 |
| sink | n/a | −1.0 | 0.3679 |

Denom $2.4048$. Weights $0.4153$ and $0.4317$. Sink mass $0.1530$. The keys do not sum to 1. That leftover is the sink: the head is allowed to attend to nothing.

$$
\text{ctx} = [0.0797,\ 0.0837,\ 0.0716,\ 0.0000]
$$

Head 2 is a summary row in the figure. Concat both heads, grouped $W_O$ (Flash: 64 heads in `o_groups = 8`):

$$
F_{\text{attn}} = [0.0415,\ 0.1235,\ -0.0035,\ 0.0665]
$$

![Layer 0 sliding-window attention for token p with RoPE, window-2 mask, and sink softmax](/assets/images/deepseek-v4-inside/fig04-layer0-swa.webp "Figure 4: layer 0 SWA, window 2. Flash layers 0-1 use compress_ratio 0.")

Flash keeps a 128-token uncompressed window even after compression starts (`sliding_window`). The first two layers never compress. Local syntax is cheap. The 1M problem is the far tokens.

## How does the residual update after attention?

$$
X_{\text{new}} = B X + C F
$$

Yellow stream 0:

$$
\begin{aligned}
BX_0 &= 0.5741\cdot[0.6,\ -0.3,\ 0.4,\ 0.2] + 0.4259\cdot[0.24,\ -0.12,\ 0.16,\ 0.08] \\
&= [0.4467,\ -0.2233,\ 0.2978,\ 0.1489] \\
C_0 F &= 1.0523\cdot F = [0.0437,\ 0.1300,\ -0.0037,\ 0.0699] \\
X_{\text{new},0} &= [0.4904,\ -0.0934,\ 0.2941,\ 0.2188]
\end{aligned}
$$

Stream 1 lands at $[0.4287,\ -0.0915,\ 0.2592,\ 0.1877]$.

![mHC residual update after attention: B mixes streams, C scales the attention output](/assets/images/deepseek-v4-inside/fig05-mhc-after-attn.webp "Figure 5: X_new = B X + C F after attention.")

## Why does layer 0 MoE hash the token id?

Config: `num_hash_layers = 3`. Paper §2.1: the first MoE layers replace a learned router with hash routing, expert from the token id, no dense FFN at the start the way V3 had.

Toy: `expert = token_id mod 4`. Second routed expert is `(e+1) mod 4` so top-2 still fires. Shared expert always on. Hash has no logits, so I fixed gates $0.7$ and $0.3$.

| Token | Id | Primary | Second |
|---|---:|---:|---:|
| d | 100 | 0 | 1 |
| e | 101 | 1 | 2 |
| e | 101 | 1 | 2 |
| **p** | **112** | **0** | **1** |
| s | 115 | 3 | 0 |
| e | 101 | 1 | 2 |
| e | 101 | 1 | 2 |
| k | 107 | 3 | 0 |

Occupancy of the hashed primary: $[2,\ 4,\ 0,\ 2]$. Expert 2 is empty. That is why later layers switch to a learned router plus `noaux_tc` bias.

![Layer 0 hash routing: token id mod 4, shared expert always on, occupancy bars](/assets/images/deepseek-v4-inside/fig06-hash-routing.webp "Figure 6: hash routing. Flash hashes the first 3 MoE layers.")

Input to the expert, after mixing the post-attention streams:

$$
h_p = [0.4874,\ -0.0973,\ 0.2933,\ 0.2159]
$$

Expert 0 is SwiGLU, intermediate 3 (Flash 2048), output clamped to $[-10,10]$ (`swiglu_limit`).

$$
\text{gate} = h W_{\text{gate}} = [0.2109,\ 0.0449,\ 0.0533]
$$

$\mathrm{SiLU}(g)=g\cdot\sigma(g)$:

| $g$ | $\sigma(g)$ | SiLU | $\times$ up | hid |
|---:|---:|---:|---:|---:|
| 0.2109 | 0.5525 | 0.1166 | 0.1520 | 0.0177 |
| 0.0449 | 0.5112 | 0.0230 | −0.0229 | −0.0005 |
| 0.0533 | 0.5133 | 0.0274 | 0.2852 | 0.0078 |

$y = \text{hid}\,W_{\text{down}} = [0.0062,\ 0.0032,\ 0.0013,\ 0.0044]$. Clamp does not bite. On FP8/FP4 training it does; that is why the limit exists (paper §4.2.3).

![SwiGLU inside expert 0 for token p: gate, up, silu, down, clamp](/assets/images/deepseek-v4-inside/fig07-swiglu.webp "Figure 7: one SwiGLU expert. Intermediate 3, Flash 2048.")

Combine: shared $+$ gated routed.

| Path | Vector |
|---|---|
| Shared | $[0.0030,\ 0.0037,\ 0.0074,\ 0.0043]$ |
| Expert 0 | $[0.0062,\ 0.0032,\ 0.0013,\ 0.0044]$ |
| Expert 1 | $[0.0068,\ 0.0029,\ -0.0004,\ 0.0039]$ |
| $0.7 e_0 + 0.3 e_1$ | $[0.0064,\ 0.0031,\ 0.0008,\ 0.0043]$ |
| $F_{\text{MoE}}$ | $[0.0094,\ 0.0068,\ 0.0082,\ 0.0085]$ |

Second mHC wrap. $X$ after layer 0, yellow:

| Stream | $X$ |
|---|---|
| 0 | $[0.4739,\ -0.0854,\ 0.2879,\ 0.2146]$ |
| 1 | $[0.4629,\ -0.0865,\ 0.2811,\ 0.2082]$ |

Flash activates 6 of 256 routed experts plus 1 shared. That is the 13B-active number sitting inside 284B total.

![MoE combine of shared and two routed experts, then mHC, X after layer 0](/assets/images/deepseek-v4-inside/fig08-moe-combine.webp "Figure 8: MoE combine and X after layer 0.")

## How does CSA compress 8 tokens into 4 KV entries?

Paper §2.3.1: every $m$ tokens become one KV entry by an overlapped weighted sum of two streams $C^a$ and $C^b$. Adjacent blocks share tokens. Then a lightning indexer keeps top-$k$ (Flash 512). Core attention is shared-KV MQA: the compressed entry is both $K$ and $V$. An uncompressed window of `sliding_window` tokens is concatenated so a query can see inside its own compression block without breaking causality.

Toy $m=2$, overlap weights $C^a=[0.5,\ 0.3]$, $C^b=[0.2]$ on the next token. Entry 2 uses tokens of entry 1. $H$ after mixing layer-0 $X$, yellow $[0.4930,\ -0.0902,\ 0.2994,\ 0.2226]$.

| Entry | Token indices | Compressed KV |
|---|---|---|
| 0 | 0, 1, 2 | $[0.3717,\ 0.3422,\ -0.0321,\ 0.0540]$ |
| 1 | 2, 3, 4 | $[0.2972,\ 0.2917,\ 0.1721,\ 0.0681]$ |
| 2 | 4, 5, 6 | $[0.1243,\ 0.4243,\ 0.1979,\ 0.1469]$ |
| 3 | 6, 7 | $[0.1889,\ 0.1770,\ 0.1481,\ -0.0288]$ |

Entry 1 contains future token `s` (position 5) relative to query $p$. Causal rule: an entry is visible only if every token in it has position $\le$ the query. Entry 1 is forbidden. The indexer can still score it. Core attention must not.

![CSA compression: 8 tokens, m=2, overlap weights Ca and Cb, four compressed KV entries](/assets/images/deepseek-v4-inside/fig09-csa-compression.webp "Figure 9: CSA compression. Entry 2 overlaps entry 1's tokens.")

> [!GOTCHA]
> Flash CSA softmax-normalizes $2m$ overlap weights per block and uses learned $C^a$, $C^b$ plus positional biases. The toy keeps three weights that sum to 1 so the grid stays countable. Same idea, not the same tensor.

RMSNorm on the Q head and on the compressed KV head, then partial RoPE on the last 2 dims (Flash last 64). Indexer scores of $p$ against the four entries, ReLU, rank:

| Entry | Score | ReLU | Causal? |
|---|---:|---:|---|
| 0 | 0.0891 | 0.0891 | yes |
| 1 | 0.1076 | 0.1076 | **no** (contains `s`) |
| 2 | 0.0923 | 0.0923 | no |
| 3 | 0.0775 | 0.0775 | no |

Top-2 by score would be 1 then 2. After the mask, only entry 0 remains. Window $n_{\text{win}}=2$ adds uncompressed tokens 3 and 4 (`e` and `p`). Keys for core attention: 1 compressed + 2 window.

![CSA pre-attention: RMSNorm, partial RoPE, lightning indexer scores, ReLU, top-2, causal mask](/assets/images/deepseek-v4-inside/fig10-csa-indexer.webp "Figure 10: indexer. Flash top-512. Toy top-2, then causality drops the future block.")

![CSA core attention for token p: selected compressed entry plus uncompressed window, MQA, sink, softmax](/assets/images/deepseek-v4-inside/fig11-csa-core.webp "Figure 11: CSA core attention. Same entry is K and V.")

Scores for $p$, sink in the denom: $[0.9980,\ 0.1329,\ 1.9828]$, weights $[0.2362,\ 0.0994,\ 0.6323]$, sink mass $0.032$. Context $[1.4511,\ 0.2906,\ -0.6491,\ -0.3722]$.

Then RoPE with position $-i$ on the last 2 dims of that output. Paper: the compressed KV already carries absolute positions, so the weighted sum would leak them unless you rotate the output by $-i$. After that, grouped $W_O$:

$$
F_{\text{CSA}} = [0.8717,\ 0.9787,\ 0.0663,\ 0.6627]
$$

Learned router on the post-CSA hidden. Affinity is $\sqrt{\mathrm{softplus}(x)}$, not V3's sigmoid (`scoring_func": "sqrtsoftplus"`). Bias $b=[0.0,\ 0.1,\ -0.1,\ 0.0]$ is added **only** for top-k selection (`topk_method": "noaux_tc"`). Gates are renormalized from the unbiased scores.

| Expert | Logit | softplus | $\sqrt{\cdot}$ | $+$ bias | Selected |
|---|---:|---:|---:|---:|---|
| 0 | 0.6466 | 1.0679 | 1.0334 | 1.0334 | yes |
| 1 | 0.0176 | 0.7020 | 0.8378 | 0.9378 | no |
| 2 | 0.3729 | 0.8969 | 0.9470 | 0.8470 | no |
| 3 | 1.0715 | 1.3661 | 1.1688 | 1.1688 | yes |

Top-2: experts 3 and 0. Gates $0.5307$ and $0.4693$.

![Output RoPE with position minus i, grouped projection, then learned sqrt-softplus router with selection bias](/assets/images/deepseek-v4-inside/fig12-rope-router.webp "Figure 12: output RoPE at -i, then learned router. Bias is for selection only.")

## What does HCA change?

Same MQA, window, RoPE, sink. Compression $m'=4$ (Flash 128), **no overlap**, **no indexer**. Dense attention over $n/m'$ entries. At toy $T=8$ that is 2 entries: tokens 0-3 and 4-7. Yellow $p$ lives in entry 0. Entry 1 still contains the future, so a causal query at $p$ only sees entry 0 plus the 2-token window.

![HCA compression m-prime=4, dense attention over two entries plus window, KV memory bars at toy and Flash scale](/assets/images/deepseek-v4-inside/fig13-hca-memory.webp "Figure 13: HCA. No indexer. Memory bars: SWA vs CSA vs HCA.")

KV **entries per query** at Flash context $1{,}048{,}576$:

| Attention | Entries the query scores | How |
|---|---:|---|
| SWA | 128 | `sliding_window` |
| CSA | $512 + 128 = 640$ | indexer top-$k$ + window |
| HCA | $8192 + 128 = 8320$ | $1{,}048{,}576/128$ dense compressed + window |
| Plain dense | $1{,}048{,}576$ | every past token |

CSA is the sparse one. HCA is the heavy shrink with no indexer. Interleaving them is the hybrid in `compress_ratios`. Paper §2.3.4: mixed BF16 on the RoPE dims and FP8 on the rest, FP4 in the indexer, and versus a BF16 GQA-8 head-dim-128 baseline the 1M KV cache is about 2%. Versus DeepSeek-V3.2, Pro at 1M is 27% of the single-token FLOPs and 10% of the KV; Flash is 10% of the FLOPs and 7% of the KV.

## How does noaux_tc move expert bias?

No auxiliary load-balancing loss. After a batch, if expert $i$ ran hot, nudge $b_i$ down so the next batch is less likely to pick it. If it ran cold, nudge $b_i$ up. The bias does not enter the gate values, only the top-k pick. A small sequence-wise balance term stops one sequence from dumping everything on one expert (paper §2.1). Hash layers do not learn this; layers 0-2 in Flash skip the router.

![noaux_tc bias update from expert load counts, hash layers versus learned layers on the 43-layer strip](/assets/images/deepseek-v4-inside/fig14-noaux-tc.webp "Figure 14: noaux_tc. Overloaded experts get bias down.")

On the toy hash layer, occupancy $[2,4,0,2]$ already shows the failure mode a learned bias is meant to fix: one expert empty, one stuffed with every `e`.

## What does the main head predict for t+1?

Final mix of the two streams into $[8\times 4]$, RMSNorm, LM head. Vocab 256 in the toy; I only score four candidate bytes: `e` (101), `k` (107), space (32), `s` (115). True next byte after $p$ is `s`.

![Final mHC mix, RMSNorm, LM head logits restricted to four candidate bytes, softmax for token p](/assets/images/deepseek-v4-inside/fig15-lm-head.webp "Figure 15: main head, token t+1. Inference uses this head only.")

## Where does the extra t+2 head get its input?

MTP depth 1, same wiring as V3 (paper §2.1: "without modification"). During training, concatenate the main hidden state of $p$ with the embedding of the **true** next byte `s`, RMSNorm, a small projection, one transformer block, a second head. Inference discards this path.

![MTP block: main hidden of p concatenated with embedding of true next byte, one block, logits for t+2](/assets/images/deepseek-v4-inside/fig16-mtp.webp "Figure 16: sequential MTP. True t+1 embedding is teacher-forced.")

## What is the loss, and who updates which weights?

V3 used $L = L_{\text{lm}} + 0.3\, L_{\text{mtp}}$. V4 keeps that MTP setup. $L_{\text{lm}}$ is cross entropy on $t+1$ (`s` for this $p$). $L_{\text{mtp}}$ is cross entropy on $t+2$ (`e`). See [cross entropy](/blog/cross-entropy-loss-explained/) if the softmax-NLL step is rusty.

Muon (paper Algorithm 1, §2.4) updates the 2D weight matrices: Newton-Schulz orthogonalization of the momentum, then a scaled step. Embeddings, RMSNorm gains, and mHC static gates stay on AdamW. Training is FP8/FP4 on clusters, $32$T+ tokens, then YaRN from 64K `original_max_position_embeddings` to $1{,}048{,}576$ (`rope_scaling.type = yarn`, factor 16).

![Loss L_lm plus 0.3 L_mtp at token p, and a coloured map of Muon versus AdamW parameters](/assets/images/deepseek-v4-inside/fig17-loss-optimizers.webp "Figure 17: MTP-weighted loss and optimizer split. Muon on 2D weights.")

## What is stored at generate time?

Main head only. The KV cache is compressed entries, not per-token $K,V$ for every layer. After 8 toy tokens: 4 CSA entries, 2 HCA entries. The next byte grows the cache by a fraction of an entry ($1/m$ or $1/m'$), not by a full head-dim vector per layer the way a vanilla transformer does. Flash also keeps the 128-token uncompressed window.

![Generation step: main head sample, CSA and HCA compressed KV cache after 8 tokens, cache growing by a fraction of an entry](/assets/images/deepseek-v4-inside/fig18-generate-cache.webp "Figure 18: generate. Cache is compressed entries plus a short window.")

That is the 1M trick in one picture: you are not storing a million keys per head. You are storing $n/4$ or $n/128$ mixed slots, plus 128 locals, plus (for CSA) a top-512 gather.

## Can I see p's numbers on one page?

![One-page memory poster of the pipeline with yellow-token values, five core equations, and Flash-scale numbers](/assets/images/deepseek-v4-inside/fig19-memory-poster.webp "Figure 19: one-page poster. Five equations and the Flash knobs that matter.")

Five equations I actually used:

$$
\begin{aligned}
X_{l+1} &= B_l X_l + C_l F_l(A_l X_l) \\
\text{affinity} &= \sqrt{\mathrm{softplus}(x)} \\
C^{\text{Comp}}_i &= \text{overlapped weighted sum of block }i \\
s_{ij} &= \frac{\exp(z_{ij})}{\sum_k\exp(z_{ik})+\exp(z')} \\
L &= L_{\text{lm}} + \lambda L_{\text{mtp}}
\end{aligned}
$$

Eight Flash numbers worth keeping: 43 layers, $d=4096$, $m=4$, $m'=128$, indexer 512, window 128, 256 routed / top-6 / 1 shared, context $1{,}048{,}576$.

<details markdown="1">
<summary>Shapes I would check first if I implemented the toy (batch 1, T = 8)</summary>

1. Token ids `[8]`, embedding `[8, 4]`.
2. Residual $X$ `[2, 8, 4]`.
3. $A X$ `[8, 4]`.
4. Per-head $Q$ `[8, 4]`; CSA/HCA compressed KV `[T/m, 4]` or `[T/m', 4]`.
5. SWA scores `[8, window]` with window 2.
6. CSA indexer scores `[8, 4]` then a top-2 index tensor.
7. MoE gates `[8, 2]` plus a shared output `[8, 4]`.
8. Expert hidden `[8, 3]` inside SwiGLU.
9. LM logits toy `[8, 256]`; MTP hidden concat `[8, 8]` before the projection back to 4.
10. Cache after 8 tokens: CSA length 4, HCA length 2, window 2.

</details>

**Read next:** the [June V4 engineering notes](/blog/deepseek-v4-engineering-explained/) for the verification checklist and the $n^2$ cost, then [how attention works](/blog/attention-in-transformers-explained/) if the score-row still feels like a vibe. The paper HTML is [arXiv 2606.19348](https://arxiv.org/html/2606.19348). The file I counted is [Flash-0731 `config.json`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/config.json).

[^paper]: DeepSeek-AI, *DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence*, arXiv [2606.19348](https://arxiv.org/abs/2606.19348). Citation date 26 April 2026. Hybrid CSA/HCA §2.3, mHC §2.2, MoE and hash layers §2.1, MTP same as V3 §2.1, Muon §2.4, Flash/Pro widths in the abstract and §4.2.1. Checkpoints: [deepseek-ai/deepseek-v4](https://huggingface.co/collections/deepseek-ai/deepseek-v4).

[^config]: [deepseek-ai/DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) `config.json`. `compress_ratios`, `hc_mult`, `hc_sinkhorn_iters`, `scoring_func`, `topk_method`, `num_nextn_predict_layers`, `num_hash_layers`, `index_topk`, `sliding_window`, `swiglu_limit`, YaRN block. Hub parameter field 304180.4M for this repo; paper abstract 284B / 13B.

[^v3mtp]: DeepSeek-V3 technical report, arXiv [2412.19437](https://arxiv.org/abs/2412.19437), sequential MTP and $\lambda = 0.3$. V4 §2.1 says the MTP configuration is unchanged.