One Transformer Block Can Do the Work of Many Layers
A new approach lets a single Vision Transformer block repeat itself at different "depths," matching bigger models while using far fewer parameters.

A program that analyzes images using a single repeated processing unit instead of many unique layers, making it much smaller.
- Give it a picture to analyze
- One block runs over and over again
- It blends its parts differently each round
- You get the result quickly
Stacking Blocks or Recycling Them?
Most modern vision transformers — neural networks that process images — work like an assembly line. An image passes through a stack of many transformer blocks, each one refining the picture's features 1. Standard Vision Transformers (ViTs) typically use dozens of these blocks, and each block stores its own separate set of weights. This works well for accuracy, but it costs a lot of memory.
Researchers Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos asked a simpler question: what if you didn't need dozens of unique blocks at all? What if one block, repeated over and over, could do the same job?
One Block, Many Personalities
Their answer is a system they call reViT — a recurrent Vision Transformer with depth-programmed experts. Instead of giving each layer its own exclusive FFN (feed-forward network, the part of a transformer that processes information), reViT shares a small bank of expert FFNs and mixes them together differently at each repetition 1.
Think of it like a chef who has a limited set of core ingredients but creates entirely different dishes depending on how much of each ingredient they use. A continuous "normalized-depth" number tells the model exactly how to blend the experts at any given step, tracing a smooth path through the model's parameters as it runs 1.
Smaller But Still Strong
The results are striking. When trained from scratch on ImageNet-1k, a reViT-B/16 model reaches the same accuracy as DeiT III but stores roughly 70% fewer parameters 1. That means a much smaller model that still performs at the level of a much larger one.
When the researchers distilled knowledge from a large teacher model called DINOv2 using only output features, an 8-expert version of reViT kept nearly all of the teacher's accuracy on linear-probe tests — meaning you can easily train a simple classifier on top of its learned features 1. This distilled model also transferred well to other tasks like image segmentation and depth prediction.
One Model, Any Depth You Need
Perhaps the most practical feature is something the team calls elastic-depth training. Because the model learns a continuous trajectory through its expert mixtures, a single saved checkpoint can be run at multiple different depths simply by resampling along the same trajectory 1. You get one model that can flexibly trade speed for accuracy on the fly.
For situations where you need fixed performance, the recurrent block can also be "unrolled" into a traditional static graph, removing the need for on-the-fly mixing while keeping the same computational cost per step 1.
The work was posted to arXiv in October 2026 and represents a shift toward more efficient vision architectures that rethink how much repetition and parameter reuse can accomplish 3.
Key facts
- A single recurrent transformer block can match the accuracy of a full-depth vision encoder
- reViT-B/16 achieves DeiT III accuracy with about 70% fewer stored parameters
- An 8-expert distilled model retains nearly all of its DINOv2 teacher's accuracy
- Elastic-depth training lets one checkpoint operate at multiple depths
- Weight-space merging was identified as the strongest Mixture of Experts approach tested
Sources
If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work
Keep reading
How it worksHow Meta boosts engineering productivity by 3× using agents to propagate ad model techniquesHerald of AI
How Meta boosts engineering productivity by 3× using agents to propagate ad model techniques
Meta's new AI agent system adapts small improvements across thousands of ad models automatically, saving millions in computing costs.
How it worksA large AI model learned to fix its own brain without being retrainedHerald of AI
A large AI model learned to fix its own brain without being retrained
Researchers found a 27-billion-parameter model can edit its own internal settings to become more accurate — no human teacher needed.
How it worksAI shopping bots get more confident asking to think harder — but not any smarterHerald of AIDeni Williams from São Paulo, Brasil / Brazil, CC BY 2.0, via Wikimedia Commons
AI shopping bots get more confident asking to think harder — but not any smarter
A new study shows that making AI assistants reason longer before suggesting products only makes them sound sure of themselves, not actually more right.