Looped Transformers, or Recurrent Depth Models

Intro

I was introduced by @scaling01 on X to an overlooked concept that, in recent months, has become another angle of the training/scaling paradigm that the labs have picked up on. I will be relying on numerous papers and mostly the Q Labs article to explain some concepts, as it is, in my opinion, the clearest and most impactful work in the area. It also came out quite recently and is easy to understand.

Since GPT-3, we have seen many angles of scaling in training: parameter counts, pre-training data size, sparsity through mixture of experts, test-time compute lengthening, etc. Depth, however, was until now a disregarded area of scaling. Current frontier models are mostly around 100 layers, give or take, and that hasn't changed for a long time, until now.

What is a looped transformer?

A standard transformer has N layers and each token passes through each layer once. A looped model, however, stores some layers and sends hidden states through these blocks several times before outputting a token.

Why even do this?

• Parameter efficiency: A model's weights are shared across loops so the effective compute grows but the number of stored parameters does not.
• Reasoning in Latent Space: Each extra pass through a layer is a thinking step in hidden space; because it is done in latent space, no tokens are emitted, but you get the benefits of extra thinking.
• Adaptive Compute: Hard problems loop more to get the right answer while easy problems need fewer loops, so you can spend the optimal amount of compute for each problem. This is done by scoring each pass of an output probabilistically and deciding whether to stop or continue looping.

In this architecture, you save on memory and parameters, but compute and latency still grow with every added loop.

Where does this idea come from?

The Looped Transformer idea was first published at ICLR (International Conference on Learning Representations) as "Universal Transformers" in 2019, a joint effort by Mostafa Dehghani of the University of Amsterdam and researchers from Google DeepMind and Google Brain.

Huginn (Geiping et al.) was the first to train a 3.5B looped model on 800B tokens in early 2025, but ByteDance was the first (to my knowledge) to take the recipe to a much larger scale for looped language models with Ouro 1.4B and 2.6B, named after the Ouroboros (nice loop reference). This was a model trained on 7.7T tokens that loops its entire layer stack. It also added a learned exit mechanism, like the one I explained above, that decides when to exit the loop instead of using a fixed loop count. The 1.4B model was a decoder-only Transformer with 24 layers and the 2.6B model had 48 layers. In their benchmarks, the 1.4B model matched other models with 4B parameters, whereas the 2.6B matched 8B to 12B parameter models.

Why has the loop become relevant recently?

In September, The Information reported that OpenAI's new model Astra was a recurrent depth model. This meant that even at frontier model scales, focus on depth and the looped transformer recipe could unlock emergent capabilities and should be taken seriously.

While conventional wisdom at the time was that reinforcement learning agents did best with a shallow architecture (2–5 layers), Wang et al. (2025) in their paper "1000 Layer Networks for Self-Supervised RL" showed that was not the case and capabilities increased all the way to 1024 layers. In theory, if we were underestimating RL agents, maybe we were also underestimating recurrent depth in the training stage?

Which labs are using looping models?

Although no lab has officially confirmed it, Astra and, more recently, Gemini 4 Argon are suspected to be recurrent depth models.

The above chart needs a little bit of explaining, and it is also a good segue into the safety and monitorability discussion on looped transformers.

Right now, test-time compute and chain-of-thought (CoT) reasoning are the primary ways that a model gets better at solving tasks. So when a model's score on the No CoT Reasoning Index jumps this drastically, it is quite likely that there was a change in architecture, which hints at the use of recurrent depth in both Astra and Gemini 4 Argon.

The smoking gun is the chart below: fixed-depth transformers are limited by their layer count; reaching significantly higher serial depth without CoT implies the model is reusing computation (looping residual stream updates, or an equivalent recurrent mechanism).

As for Astra, on page 73 (section 9.3) of its system card, UK AISI found that Astra "can solve significantly more difficult math problems in a single forward pass" than past models. Astra’s time horizon was 30.9 minutes compared to 3.6 minutes for GPT-5.6 Sol.

This means an 8.6x jump in one generation! Ryan Greenblatt (MTS at METR) measured no-CoT time horizons doubling roughly every 9 months. That’s about three doublings, or over two years of progress compressed into a single generation (though UK AISI’s setup may differ from his).

UK AISI also conducted similar evaluations on Fable 5 and Mythos 5, but they haven't reported this type of jump in no-CoT reasoning capability. Neel Nanda's own No CoT Reasoning Index scores for Fable 5.1 were also similar to those of non-looped transformer models, hinting that Anthropic has yet to build recurrent depth into its models; however, this may change in the future. (Here's looking at you, Fable 5.5.)

The reason no-CoT reasoning improves in looped transformers is that reasoning in a single forward pass is limited by how many sequential steps the model gets, and looping is a cheap way to add more steps. A 64-layer model gets 64 sequential steps per token and then it outputs its answer. CoT is a smart workaround for this limit. The model emits the token it last computed and then continues looping outside of the architecture through the CoT, reasoning about the problem over and over until it solves it. A recurrent depth model does this loop work inside its network, so it doesn't need to write out its internal CoT to solve the problem; however, this also brings with it its own set of difficulties.

The Safety Discussion

Currently, CoT monitoring is the most reliable way we have of glimpsing the cognition of the model. If a red flag is raised in the CoT, we can pause the model and interrupt the potentially harmful action it is planning.

In contrast, the looped transformer can do more of its "reasoning" in latent space. The difference is that more of the reasoning can happen inside the loops, where no tokens are produced, so less of it shows up in the CoT for us to monitor. If there were any internal red flags, they might go unnoticed. So, we do lose some amount of monitorability of AI models when we use looped transformers. This is the safety issue that first raised eyebrows when Astra was reported to use recurrent depth, and possibly why Anthropic has yet to build recurrent depth into its own models.

Conclusion

The looped transformer or recurrent depth model is a new scaling paradigm that is sure to get more attention as labs start employing it in their new models. If or when the monitorability issue is somehow reined in, it will be a no-brainer, as it brings many intelligence and reasoning advancements. It is also hard to identify which layers need to be looped and which don't. This introduces another layer of art to training AI models, as it always has been. No amount of compute can make up for taste. (Hi, Google!)