---
layout: post
title: "Looped Transformers Explained: Recurrent Depth and Astra"
date: 2026-09-03 13:00:00 +0530
tags: ["Transformers", "LLMs"]
category: "ai"
description: "What is a looped transformer? How recurrent depth reuses layers, why Nanbeige 4.2 runs a 22-layer stack twice, and why that does not hide chain of thought."
keywords: "looped transformer explained, recurrent depth, OpenAI Astra architecture, Nanbeige 4.2, mixture of recursions, universal transformer, KV cache sharing, latent reasoning"
image: /assets/images/og-looped-transformers-explained.png
---

I watched [Sebastian Raschka's Astra video](https://www.youtube.com/watch?v=KT4n-z_4QJU) because the rumor makes two claims at once: that Astra has a particular architecture, and that this architecture hides chain of thought. The first claim can be checked. The second does not follow from it.

OpenAI's own [Path to Astra](https://openai.com/index/path-to-astra) post (1 September 2026) confirms Astra as an upcoming model and says it meets the Critical cybersecurity threshold under their Preparedness Framework. Separate OpenAI notes on [cyber-critical safeguards](https://openai.com/index/responding-next-frontier-critical-cyber-capabilities) describe monitors that read the model's chain of thought and can interrupt high-risk activity. Neither post says "we loop a transformer stack." That claim comes from [The Information](https://www.theinformation.com/), as Raschka quotes it. I treat the architecture as a rumor attached to a real model, and the maths as the part I can work through myself.

{% include youtube.html id="KT4n-z_4QJU" title="OpenAI Astra and Recurrent Depth / Looped Transformers" caption="Sebastian Raschka, 2 September 2026. Architecture notes on a rumor, not an OpenAI system card." %}

> [!NOTE]
> The loop is ordinary transformer blocks run more than once. If queries, keys, values, and the KV cache still feel vague, start with [how attention works](/blog/attention-in-transformers-explained/) and the [full transformer forward/backward pass](/blog/transformer-from-scratch-forward-backward-math/). I use those mechanics here.

## What did the rumor claim?

Raschka reads The Information as saying two things:

1. Astra uses "recurrent depth" or "loop transformers," and this lets the model improve answers by processing the same text multiple times.
2. Unlike current frontier models that write out how they are thinking, the technique obscures some or all of that reasoning, so humans cannot easily read the steps.

Claim 1 describes an architecture that already exists in public models. Claim 2 does not follow from claim 1. I cover the mechanism first and the safety argument after it.

One correction up front: the loop does not re-tokenize the prompt or reread the paragraph. After the embedding lookup, the model holds hidden states, and those vectors go through the same weight matrices a second time. The weights are the same and the activations are new.

## What is a looped transformer?

A standard decoder is a stack of distinct blocks. Layer 1 has its own $W_Q, W_K, W_V, W_O$ and MLP. Layer 2 has a different set. You apply each once, then the unembedding.

A looped transformer (also called recurrent depth, or looped depth sharing) keeps fewer unique blocks and applies that stack more than once. If $L$ unique layers run for $R$ passes, you get $LR$ block applications and $L$ sets of weights.

```mermaid
flowchart LR
  X["token embeddings"] --> S["22 unique blocks"]
  S -->|"pass 2, same weights"| S
  S --> Y["RMSNorm + unembedding"]
```

Nanbeige 4.2-3B is the public example I can measure. Its [config](https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/config.json) states the loop directly:

```json
{
  "hidden_size": 3072,
  "num_hidden_layers": 22,
  "num_loops": 2,
  "num_attention_heads": 48,
  "num_key_value_heads": 8,
  "head_dim": 128,
  "intermediate_size": 10752,
  "vocab_size": 166144,
  "max_position_embeddings": 262144,
  "tie_word_embeddings": false
}
```
{: data-title="Nanbeige4.2-3B/config.json"}

The model has 22 unique decoder layers and `num_loops: 2`. Hidden states finish the stack, then go through the same 22 layers again. Effective depth is 44, while the unique transformer weights stay at 22 layers' worth.

The [technical report](https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/Nanbeige42_report.pdf) (Nanbeige LLM Lab, Boss Zhipin) says they pretrained this from scratch on 28T tokens. The card rounds the size as 4B total parameters, 3B non-embedding.[^nb-report]

> [!GOTCHA]
> The video calls Nanbeige the first frontier-level open-weight model out of India. The PDF author line is Nanbeige LLM Lab, Boss Zhipin. I am ignoring the geography and keeping the layer counts.

## How much depth do you buy without duplicating weights?

I leave embeddings out for now, the way the report does when it talks about "capacity without adding parameters." Each Nanbeige block is GQA attention plus a SwiGLU MLP.

Query heads are $48 \times 128 = 6144$ wide, while the residual stream is $3072$. KV is $8 \times 128 = 1024$ wide. One layer's attention matrices:

$$
\begin{aligned}
Q &: 3072 \times 6144 = 18{,}874{,}368 \\
K &: 3072 \times 1024 = 3{,}145{,}728 \\
V &: 3072 \times 1024 = 3{,}145{,}728 \\
O &: 6144 \times 3072 = 18{,}874{,}368
\end{aligned}
$$

$$
18{,}874{,}368 + 3{,}145{,}728 + 3{,}145{,}728 + 18{,}874{,}368 = 44{,}040{,}192
$$

SwiGLU is three $3072 \times 10752$ maps:

$$
3 \times 3072 \times 10752 = 99{,}090{,}432
$$

Two RMSNorm scales add $2 \times 3072 = 6144$. Per unique layer, about $143.1$ million parameters. Twenty-two unique layers:

$$
22 \times 143{,}136{,}768 = 3{,}149{,}008{,}896
$$

That is the "3B non-embedding" number, rounded. Untied embeddings and lm_head are $166144 \times 3072$ twice, about $1.02$B more, which is how you get a card that says 4B total.

Now duplicate the stack into 44 *unique* layers instead of looping:

| Setup | Unique blocks | Block applications | Block parameters (approx.) |
|---|---:|---:|---:|
| Standard 22-layer | 22 | 22 | 3.15B |
| Looped 22 × 2 (Nanbeige) | 22 | 44 | 3.15B |
| Standard 44-layer | 44 | 44 | 6.30B |

Memory to *store* the transformer weights stays near the 22-layer model. Training and decoding still pay for 44 block applications: the activations go through 44 attention-plus-MLP steps, and backprop walks 44 applications. Looping saves parameters, not FLOPs in the stack.

Embedding and unembedding are the exception. They are applied once per token either way, so "almost 2× compute" holds for the blocks but not for the whole model.

I added more combinations of unique layers and passes. The highlighted row is Nanbeige.

{% include fig.html src="looped-transformers-explained/depth-setups.svg" alt="Table of unique layers, passes, block parameters, block applications and FLOPs against 22 layers. The 22 by 2 row has 3.15B parameters, 44 applications and 2.00 times the FLOPs. A 44-layer model with 1 pass has 6.30B parameters and 2.00 times the FLOPs. A 22-layer stack with 4 passes has 88 applications and 4.00 times." caption="Passes multiply FLOPs but not parameters. Extra unique layers multiply both." %}

## Why train the loop from scratch?

Section 2.1 of the Nanbeige report compares two recipes.

**Upcycle.** Train a normal 22-layer model. After pretraining, start looping the finished stack.

**From scratch.** Put `num_loops: 2` in the architecture on day one so every pretraining token sees two passes.

They write that from-scratch "performs significantly better." That matches what I would expect from other architecture changes. Swap GELU for SwiGLU on a finished checkpoint and the next-token distribution shifts, because the model spent 28T tokens learning the old function. A second pass through the same weights is a bigger change than a new activation. The report does not publish the full with-or-without-loop ablation (they note the cost), so I treat their comparison as an internal result, not a public table.

## Why two passes, not four?

Same section: two passes were the best trade-off they measured. Relative to a standard transformer, two passes "retain approximately 75% of the token efficiency" while still adding capacity. Extra passes added little, slowed training, and made optimization less stable.

Token efficiency here means how far the loss drops per token seen. A second pass costs almost another full stack of FLOPs per token, but it does not supply another full stack of unique features. For them, two passes was the last point where the trade still looked worth it. I have not rerun the sweep.

## Why keep a unique KV cache per pass?

Each pass needs its own cache, and this surprised me the first time, although it follows directly from the maths.

[Attention](/blog/attention-in-transformers-explained/) at a block is $Q K^\top$ from **that block's input**, then a weighted sum of **that block's** $V$. Pass 1, layer 1 sees the post-embedding residual. Pass 2, layer 1 sees the residual after 22 blocks. The $W_K$ and $W_V$ are the same, but $x$ differs, so $K = x W_K$ is a different matrix.

Sharing one KV tensor across both passes would treat two different residuals as if they were the same. Nanbeige tried it: halving the KV cache underperformed the full, non-shared cache, so they kept the unique cache.

I used the same KV formula as in the [DeepSeek long-context notes](/blog/deepseek-v4-engineering-explained/):

$$
M_{KV} = 2\, L_{\text{applied}}\, h_{kv}\, d_h\, n\, b
$$

Nanbeige: $h_{kv} = 8$, $d_h = 128$, $b = 2$ (bf16). Per token, per applied layer:

$$
2 \times 8 \times 128 \times 2 = 4096 \text{ bytes}
$$

At the config max $n = 262144$:

| KV policy | $L_{\text{applied}}$ | Bytes per token | Cache at 262,144 tokens |
|---|---:|---:|---:|
| Unique per pass (what they shipped) | 44 | 180,224 | 44 GiB |
| Shared across passes | 22 | 90,112 | 22 GiB |

The 44 GiB figure is $176\,\text{KiB/token} \times 262144$. That counts only the KV tensors, not weights or activations. Looping does not give a free 44-layer model: it costs 44-layer compute and, to keep the accuracy they measured, 44-layer KV.

I computed the cache for both policies at three context lengths. The 262,144 row is the table above.

{% include fig-pair.html src_a="looped-transformers-explained/kv-unique.svg" alt_a="Bars of KV cache for a unique cache per pass, 44 layers. 1.38 GiB at 8,192 tokens, 5.50 GiB at 32,768 and 44.00 GiB at 262,144." caption_a="Unique cache per pass: 44 GiB at 262,144 tokens." src_b="looped-transformers-explained/kv-shared.svg" alt_b="Bars of KV cache for a shared cache, 22 layers. 0.69 GiB at 8,192 tokens, 2.75 GiB at 32,768 and 22.00 GiB at 262,144." caption_b="Shared cache: 22 GiB at 262,144 tokens." caption="Sharing halves the cache. Nanbeige found that it underperformed." %}

> [!IMPORTANT]
> Parameter tying and KV tying are different decisions. Nanbeige ties the **weights** across passes and refuses to tie the **cache**.

## What does Mixture-of-Recursions add?

Nanbeige uses a fixed loop: every token takes both passes.

[Mixture-of-Recursions](https://arxiv.org/abs/2507.10524) (Bae, Kim, Bayat, et al., 2025) puts a router on that loop, in the same spirit as a mixture-of-experts gate, except the "experts" are extra trips through one shared stack. Easy tokens leave after one pass, and harder tokens go around again.

They train this at four sizes, 135M, 360M, 730M, and 1.7B, with three recursion steps, so the recursive variants hold about one-third as many unique layer weights as the vanilla baseline at a given FLOP budget.[^mor]

```mermaid
flowchart TB
  A["all tokens, pass 1<br/>shared stack"] --> R1["router"]
  R1 -->|"exit"| OUT["unembedding"]
  R1 -->|"continue"| B["subset, pass 2<br/>same weights"]
  B --> R2["router"]
  R2 -->|"exit"| OUT
  R2 -->|"continue"| C["fewer tokens, pass 3"]
  C --> OUT
```

The paper has two router designs: a single router that assigns a depth, or a router at each loop that decides whether this token continues. They do the same job with different bookkeeping.

Attention becomes harder to define. At pass 3, some earlier tokens have already left, so you can:

- attend only to tokens that are still alive at that depth (cheaper, smaller cache, less context)
- attend to some stored KV from tokens that exited (more complete, more memory)

This is the same trade as in sliding-window or sparse attention: full context is somewhat more accurate and much more expensive. The paper also tries sharing KV across recursion depth and reports a quality drop, the same direction as Nanbeige.

On their isoFLOP plots, vanilla sometimes beats a *fixed* recursive transformer, especially at the small end. MoR's per-token routing is the setting where they report the recursive idea pulling ahead as scale grows. I do not copy axis values off a video thumbnail: the paper is the source, and the video walks through its figures.

I drew two cases with three passes. On the left every token goes through all three, which is a fixed recursive model. On the right half of the tokens continue after each pass.

{% include fig-pair.html src_a="looped-transformers-explained/mor-fixed.svg" alt_a="Bars of the share of tokens in each pass. All three passes hold 100 percent. That is 3.00 passes per token and 100 percent of the compute of 3 full passes." caption_a="Fixed loop: 3.00 passes per token, 100% of the compute." src_b="looped-transformers-explained/mor-routed.svg" alt_b="Bars of the share of tokens in each pass: 100, 50 and 25 percent. That is 1.75 passes per token and 58 percent of the compute of 3 full passes." caption_b="Routed: 1.75 passes per token, 58% of the compute." caption="The right case is an example I chose, not a number from the paper." %}

## How far back does this go?

Raschka dates the lineage to the Universal Transformer. The video gives the year as 2008, but the paper is [Dehghani, Gouws, Vinyals, Uszkoreit, Kaiser, 2018](https://arxiv.org/abs/1807.03819) (ICLR 2019).[^ut]

A vanilla transformer is a fixed stack of *distinct* layers. A Universal Transformer revises every position's vector with a **shared** self-attention plus transition, $T$ times, so depth is the number of revisions, not the number of unique blocks. It also adds Adaptive Computation Time so each position can halt early. Nanbeige's "same 22 layers, twice" is a coarse, modern-LLM version of that idea: a chunk of layers as the recurrent unit, $T = 2$, no per-token halt. MoR restores the halt as a learned router.

The 2025 latent-reasoning paper in the same family is Geiping, McLeish, Jain, et al., [Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach](https://arxiv.org/abs/2502.05171). They train a 3.5B model (Huginn-0125) on 800B tokens with a recurrent block that can unroll at test time, up to a compute load they compare to a 50B dense model, without writing a long chain of thought to buy that compute.[^huginn] Their diagram adds one more step: the original embedding is injected back into the recurrent state (heavy residual reuse of the input), on top of looping the hidden state. This is the same family as the Universal Transformer: more FLOPs at a fixed parameter count.

```mermaid
flowchart LR
  E["input embedding e"] --> R["recurrent block"]
  H["latent state s"] --> R
  R --> H
  E -.-> R
  H --> U["unembedding"]
```

## Does looping hide chain of thought?

This is the claim I came for, and I do not accept it from the architecture alone.

Reasoning models spend extra inference compute by emitting extra tokens: scratch work in text. Deeper *internal* computation (more unique layers, or more loops of the same layers) spends extra compute **before** the next token is written. These are separate budgets. Treating them as one is how "loop transformer" turns into "you can no longer read the model."

One model family already shows the difference. GPT 5.6 Luna and GPT 5.6 Sol are different sizes, and Sol can do more work in hidden states per token. That does not mean Sol stopped using a reasoning trace. It might need a shorter trace for the same question, and I still watch it write.

Looping is the same kind of extra internal work, bought by reusing matrices instead of adding new ones. A model that is better at the problem might emit fewer intermediate tokens, but every transformer layer has always done some work in hidden activations. Path to Astra points the other way: OpenAI is investing in chain-of-thought monitors on this model, which would make little sense if the loop had removed the transcript.

The version that stuck with me comes from maths exams. When I am tired and underprepared I fill pages, strike through and restart. When I am sharp I write less, but I still write. A shorter derivation is still a derivation.

Could some *other* technique, such as latent-only recurrence pushed hard with no text trace, make monitoring worse? Yes. Geiping et al. say explicitly that their aim is reasoning in continuous state rather than in tokens. That is a design choice. It is not what `num_loops: 2` on a 22-layer Nanbeige stack does by itself. If The Information describes a different method from a looped transformer, then the architecture name in the article is carrying too much of the claim.

## Is this recursive self-improvement?

No. Recursive self-improvement is a model editing its own training, tools, or weights so the next version is stronger. A looped transformer is a for-loop around `block(x, W)` with $W$ frozen during inference: the same parameters applied more times. It is a way to scale compute, and the idea dates to the 2018 Universal Transformer.

I will take the Astra architecture claim seriously when it appears in a system card, a paper, or a config. Until then the public objects are Nanbeige's 22 × 2, MoR's router, Huginn's test-time unroll, and the 2018 Universal Transformer.

**Watch:** [OpenAI Astra and Recurrent Depth / Looped Transformers](https://www.youtube.com/watch?v=KT4n-z_4QJU) (Raschka). His [write-up](https://sebastianraschka.com/blog/2026/openai-astra-looped-transformers.html) and [looped-depth gallery page](https://sebastianraschka.com/llm-architecture-gallery/looped-depth-sharing) have the diagrams. OpenAI's [Path to Astra](https://openai.com/index/path-to-astra) is the primary source for what they will currently say about the model.

[^nb-report]: Nanbeige LLM Lab. 2026. [Nanbeige4.2-3B: Unlocking Agentic Capabilities in a Compact Model](https://huggingface.co/Nanbeige/Nanbeige4.2-3B/blob/main/Nanbeige42_report.pdf). §2.1: from-scratch loop vs upcycling; two passes ~75% token efficiency; KV sharing halves cache and underperforms. Abstract: 3B non-embedding, 28T tokens, Looped Transformer.
[^mor]: Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, Se-Young Yun. 2025. [Mixture-of-Recursions](https://arxiv.org/abs/2507.10524). 135M / 360M / 730M / 1.7B; three recursions; expert-choice vs token-choice routers.
[^ut]: Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser. 2018. [Universal Transformers](https://arxiv.org/abs/1807.03819). ICLR 2019. Shared transition across depth; ACT halt per position.
[^huginn]: Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Tom Goldstein. 2025. [Scaling up Test-Time Compute with Latent Reasoning](https://arxiv.org/abs/2502.05171). Huginn-0125, 3.5B parameters, 800B tokens; [model](https://huggingface.co/tomg-group-umd/huginn-0125).
