I sat down with Sebastian Raschka’s Astra video because the rumor was doing two jobs at once: naming an architecture, and claiming that architecture hides chain of thought. The first job is checkable. The second is a jump.
OpenAI’s own Path to Astra post (1 September 2026) confirms Astra as an upcoming model and says it meets the Critical cybersecurity threshold under their Preparedness Framework. Separate OpenAI notes on cyber-critical safeguards describe monitors that read the model’s chain of thought and can interrupt high-risk activity. What none of those posts say is “we loop a transformer stack.” The loop claim is The Information, as Raschka quotes it. I am treating the architecture as a rumor attached to a real model, and the math as the part I can actually work.
[!NOTE] The loop is ordinary transformer blocks run more than once. If queries, keys, values, and the KV cache still feel like three matrices and a vibe, start with how attention works and the full transformer forward/backward pass. I use those mechanics here.
What did the rumor claim?
Raschka reads The Information as saying two things:
- Astra uses “recurrent depth” or “loop transformers,” and this lets the model improve answers by processing the same text multiple times.
- Unlike current frontier models that write out how they are thinking, the technique obscures some or all of that reasoning, so humans cannot easily read the steps.
Claim 1 is a description of an architecture that already exists in public models. Claim 2 does not follow from claim 1. I want the mechanism on the page before the safety argument.
One wording fix up front: the loop does not re-tokenize the prompt and “reread the paragraph.” After the embedding lookup, you have hidden states. Those vectors go through the same weight matrices a second time. Same weights, new activations.
What is a looped transformer?
A standard decoder is a stack of distinct blocks. Layer 1 has its own $W_Q, W_K, W_V, W_O$ and MLP. Layer 2 has a different set. You apply each once, then the unembedding.
A looped transformer (also called recurrent depth, or looped depth sharing) keeps fewer unique blocks and applies that stack more than once. If $L$ unique layers run for $R$ passes, you get $LR$ block applications and $L$ sets of weights.
flowchart LR
X["token embeddings"] --> S["22 unique blocks"]
S -->|"pass 2, same weights"| S
S --> Y["RMSNorm + unembedding"]
Nanbeige 4.2-3B is the public example I can measure. The config is not coy:
{
"hidden_size": 3072,
"num_hidden_layers": 22,
"num_loops": 2,
"num_attention_heads": 48,
"num_key_value_heads": 8,
"head_dim": 128,
"intermediate_size": 10752,
"vocab_size": 166144,
"max_position_embeddings": 262144,
"tie_word_embeddings": false
}
Twenty-two unique decoder layers. num_loops: 2. Hidden states finish the stack, then go through those same 22 weights again. Effective depth 44. Unique transformer weights stay at 22 layers’ worth.
The technical report (Nanbeige LLM Lab, Boss Zhipin) says they pretrained this from scratch on 28T tokens. The card rounds the size as 4B total parameters, 3B non-embedding.1
[!GOTCHA] The video calls Nanbeige the first frontier-level open-weight model out of India. The PDF author line is Nanbeige LLM Lab, Boss Zhipin. I am ignoring the geography and keeping the layer counts.
How much depth do you buy without duplicating weights?
Ignore embeddings for a second, the way the report does when it talks about “capacity without adding parameters.” Each Nanbeige block is GQA attention plus a SwiGLU MLP.
Query heads are $48 \times 128 = 6144$ wide, while the residual stream is $3072$. KV is $8 \times 128 = 1024$ wide. One layer’s attention matrices:
SwiGLU is three $3072 \times 10752$ maps:
Two RMSNorm scales add $2 \times 3072 = 6144$. Per unique layer, about $143.1$ million parameters. Twenty-two unique layers:
That is the “3B non-embedding” number, rounded. Untied embeddings and lm_head are $166144 \times 3072$ twice, about $1.02$B more, which is how you get a card that says 4B total.
Now duplicate the stack into 44 unique layers instead of looping:
| Setup | Unique blocks | Block applications | Block parameters (approx.) |
|---|---|---|---|
| Standard 22-layer | 22 | 22 | 3.15B |
| Looped 22 × 2 (Nanbeige) | 22 | 44 | 3.15B |
| Standard 44-layer | 44 | 44 | 6.30B |
RAM to store the transformer weights stays near the 22-layer model. Training and decode still pay for 44 block applications: the activations still go through 44 attention-plus-MLP steps, and backprop still walks 44 applications. You saved parameters. You did not save FLOPs in the stack.
Embedding and unembedding are the large exception. They are applied once per token either way, so “almost 2× compute” is the right slogan for the blocks and the wrong slogan for the whole model.
Why train the loop from scratch?
Section 2.1 of the Nanbeige report compares two recipes.
Upcycle. Train a normal 22-layer model. After pretraining, start looping the finished stack.
From scratch. Put num_loops: 2 in the architecture on day one so every pretraining token sees two passes.
They write that from-scratch “performs significantly better.” That matches what I would expect from every other tiny architecture change. Swap GELU for SwiGLU on a finished checkpoint and the next-token distribution moves; the model spent 28T tokens learning the old map. A second pass through the same weights is a bigger change than an activation. The report does not publish the full with-or-without-loop ablation (they note the cost), so I am taking their comparison as an internal result, not a public table.
Why two passes, not four?
Same section: two passes were the best trade-off they measured. Relative to a standard transformer, two passes “retain approximately 75% of the token efficiency” while still adding capacity. Extra passes added little, slowed training, and made optimization less stable.
Token efficiency here means how far the loss drops per token seen. You pay almost another full stack of FLOPs per token. You do not get another full stack of unique features. Two was where that curve still looked worth it for them. I have not rerun the sweep.
Why keep a unique KV cache per pass?
This is the part that surprised me the first time, and it shouldn’t have.
Attention at a block is $Q K^\top$ from that block’s input, then a weighted sum of that block’s $V$. Pass 1, layer 1 sees the post-embedding residual. Pass 2, layer 1 sees the residual after 22 blocks. Same $W_K$ and $W_V$. Different $x$. So $K = x W_K$ is a different matrix.
Sharing one KV tensor across both passes is the same kind of cheat as tying two layers that are not actually seeing the same residual. Nanbeige tried it. Cutting the KV cache in half underperformed the full, non-shared cache. They kept the unique cache.
I used the same KV formula as in the DeepSeek long-context notes:
Nanbeige: $h_{kv} = 8$, $d_h = 128$, $b = 2$ (bf16). Per token, per applied layer:
At the config max $n = 262144$:
| KV policy | $L_{\text{applied}}$ | Bytes per token | Cache at 262,144 tokens |
|---|---|---|---|
| Unique per pass (what they shipped) | 44 | 180,224 | 44 GiB |
| Shared across passes | 22 | 90,112 | 22 GiB |
The 44 GiB figure is $176\,\text{KiB/token} \times 262144$. That is only the KV tensors, not weights or activations. Looping did not give them a free 44-layer model. It gave them 44-layer compute and, if they want the accuracy they measured, 44-layer KV.
[!IMPORTANT] Parameter tying and KV tying are different decisions. Nanbeige ties the weights across passes and refuses to tie the cache.
What does Mixture-of-Recursions add?
Nanbeige is the blunt version: every token takes both passes.
Mixture-of-Recursions (Bae, Kim, Bayat, et al., 2025) puts a router on that loop, in the same spirit as a mixture-of-experts gate, except the “experts” are extra trips through one shared stack. Easy tokens leave after one pass. Harder tokens go around again.
They train this at four sizes, 135M, 360M, 730M, and 1.7B, with three recursion steps, so the recursive variants hold about one-third as many unique layer weights as the vanilla baseline at a given FLOP budget.2
flowchart TB
A["all tokens, pass 1<br/>shared stack"] --> R1["router"]
R1 -->|"exit"| OUT["unembedding"]
R1 -->|"continue"| B["subset, pass 2<br/>same weights"]
B --> R2["router"]
R2 -->|"exit"| OUT
R2 -->|"continue"| C["fewer tokens, pass 3"]
C --> OUT
Two router designs in the paper: a single router that assigns a depth, or a router at each loop that decides whether this token continues. Same job, different bookkeeping.
Attention gets messy. At pass 3, some earlier tokens already left. You can:
- attend only to tokens that are still alive at that depth (cheaper, smaller cache, less context)
- attend to some stored KV from tokens that exited (more complete, more memory)
That is the same family of trade as sliding-window or sparse attention: full context is a little more accurate and a lot more expensive. The paper also tries sharing KV across recursion depth and reports a quality drop, same direction as Nanbeige.
On their isoFLOP plots, vanilla sometimes beats a fixed recursive transformer, especially at the small end. MoR’s per-token routing is the setting where they report the recursive idea pulling ahead as scale grows. I am not copying axis values off a thumbnail. The paper is the source; the video is a walkthrough of the figures.
How far back does this go?
Raschka dates the lineage to the Universal Transformer. The year in the video comes out as 2008. The paper is Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser, 2018 (ICLR 2019).3
A vanilla transformer is a fixed stack of distinct layers. A Universal Transformer revises every position’s vector with a shared self-attention plus transition, $T$ times. Depth is the number of revisions, not the number of unique blocks. They also add Adaptive Computation Time so each position can halt early. Nanbeige’s “same 22 layers, twice” is a coarse, modern-LLM version of that idea: a chunk of layers as the recurrent unit, $T = 2$, no per-token halt. MoR puts the halt back, as a learned router.
The 2025 latent-reasoning paper in the same family is Geiping, McLeish, Jain, et al., Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. They train a 3.5B model (Huginn-0125) on 800B tokens with a recurrent block that can unroll at test time, up to a compute load they compare to a 50B dense model, without writing a long chain of thought to buy that compute.4 The extra trick in their diagram is injecting the original embedding back into the recurrent state (heavy residual reuse of the input), on top of looping the hidden state. Same family as the Universal Transformer: more FLOPs at a fixed parameter count.
flowchart LR
E["input embedding e"] --> R["recurrent block"]
H["latent state s"] --> R
R --> H
E -.-> R
H --> U["unembedding"]
Does looping hide chain of thought?
This is the claim I came for, and it is the one I do not buy from the architecture alone.
Reasoning models spend extra inference compute by emitting extra tokens: scratch work in text. Deeper internal computation (more unique layers, or more loops of the same layers) spends extra compute before the next token is written. Those are different ledgers. Mixing them up is how “loop transformer” turns into “you can no longer read the model.”
A useful comparison is already inside one model family. GPT 5.6 Luna and GPT 5.6 Sol are different sizes. Sol can do more work in hidden states per token. That does not mean Sol stopped using a reasoning trace. It might mean Sol needs a shorter notepad for the same exam question. I still watch it write.
Looping is the same kind of extra internal work, purchased with reuse instead of with new matrices. A model that is better at the problem might emit fewer intermediate tokens. Hidden activations already did some work in every transformer layer that ever existed. Path to Astra, if anything, points the other way: they are investing in CoT monitors on this model, which is a strange thing to do if you believed the loop had deleted the transcript.
The math-exam version, because it is the one that stuck: if I am tired and underprepared I fill pages, strike through, restart. If I am sharp I write less, and I still write. A shorter derivation is still a derivation.
Could some other technique (latent-only recurrence pushed hard, no text trace) make monitoring worse? Yes. Geiping et al. are explicit that their point is reasoning in continuous state rather than in tokens. That is a design choice you can make. It is not what num_loops: 2 on a 22-layer Nanbeige stack does by itself. If The Information is describing a different method than “looped transformer,” the architecture name in the article is doing too much work.
Is this recursive self-improvement?
No. Recursive self-improvement is a model editing its own training, tools, or weights so the next version is stronger. A looped transformer is a for-loop around block(x, W) with $W$ frozen during inference. Same parameters, more applications. Scaling trick. Old paper.
I will start taking the Astra architecture claim seriously when it shows up in a system card, a paper, or a config. Until then the public objects are Nanbeige’s 22 × 2, MoR’s router, Huginn’s test-time unroll, and a 2018 Universal Transformer.
Watch: OpenAI Astra and Recurrent Depth / Looped Transformers (Raschka). His write-up and looped-depth gallery page have the diagrams. OpenAI’s Path to Astra is the primary source for what they will currently say about the model.
-
Nanbeige LLM Lab. 2026. Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model. §2.1: from-scratch loop vs upcycling; two passes ~75% token efficiency; KV sharing halves cache and underperforms. Abstract: 3B non-embedding, 28T tokens, Looped Transformer. ↩
-
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun. 2025. Mixture-of-Recursions. 135M / 360M / 730M / 1.7B; three recursions; expert-choice vs token-choice routers. ↩
-
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser. 2018. Universal Transformers. ICLR 2019. Shared transition across depth; ACT halt per position. ↩
-
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein. 2025. Scaling up Test-Time Compute with Latent Reasoning. Huginn-0125, 3.5B parameters, 800B tokens; model. ↩
Support the writing
If this post helped, a coffee keeps the deep dives coming.
Buy me a coffee