How Language Models Slowly Forget the Start of Their Prompts
A new theorem shows why one kind of Transformer model, under stated conditions, gradually loses track of its prompt as it generates more text.

- You type a detailed prompt with specific instructions
- The AI generates text token by token
- Prompt influence fades as more tokens are produced
- Later writing depends more on recent text than the start
Forgetting Is Built In
A new research paper shows that language models naturally lose track of their early prompt instructions over time. The finding, published on arXiv by Shuyang Xiang, gives a name to a behavior researchers have long observed: the farther a model gets into generating text, the less it carries information from the very beginning of its prompt.
The paper calls this the “conditional no-hair theorem,” borrowing a term from physics. In astrophysics, the no-hair theorem says black holes can be described by only a few properties, losing all other details about what fell in. Here, the idea is similar: given the text a model has already generated, the original prompt’s influence fades away.
Measuring How Fast Models Forget
The researchers compared two different prompts that were followed by the exact same continuation text. Since the later output was identical, any difference in the model’s internal states had to come from the prompt alone. They measured how that difference shrank as more tokens were generated.
They found the gap between the two internal states decays roughly as a power law — meaning it shrinks like one divided by t to some power, where t is the number of tokens generated. A higher power means faster forgetting 1.
Trained models, like the ones used in everyday AI applications, forget relatively quickly, with a decay rate around 0.55 to 0.7. Untrained models start out with a much slower rate, around 0.2 to 0.5, meaning they hold on to prompt information longer. The paper also proves a mathematical limit on how much of the prompt can remain, depending on prompt length and model depth 1. The proof has conditions: it holds for transformers of finite depth of one common kind (pre-LN softmax transformers) whose attention scores stay bounded, and only when the activations entering each normalisation step stay away from zero . The measurements compare two prompts followed by the same continuation, so they describe how the trace of a prompt fades in that setting, not everything a chatbot keeps track of in a conversation.
Why It Happens
The mechanism is geometric. As the context window fills up, the direct attention the model pays to the original prompt gets diluted across an ever-growing amount of text. Later tokens act as relay runners, passing along whatever they picked up from earlier tokens — but this relay can only slow the forgetting by a small logarithmic amount, not stop it entirely.
To test this idea, the team built a simplified linear model that can be solved exactly. It reproduced the same slowing decay seen in real untrained networks and confirmed the mathematical bound cannot be improved.
When the researchers added a logit bias to cancel out the dilution effect, they found forgetting slowed down early in training and actually reversed after further training. This suggests that dilution is both necessary and sufficient for the forgetting they measured 1.
What Survives Moves to the Text
The key implication is that any property of the prompt that “survives” — such as the language it was written in — cannot stay encoded in the model’s internal states. It has to be carried forward by the generated text itself.
In the paper's terms, the further a model of this kind writes, the more what it keeps of its prompt has to live in the text it has already produced.
Corrections and updates
- Corrected on 20 October 2026. An earlier version did not say under which conditions the theorem is proved: only for transformers of finite depth of one kind (pre-LN softmax transformers) whose attention scores stay bounded, and when one more condition on their activations holds. It also said the paper explains why long texts drift, which the paper does not say. The conditions have been added and that claim removed.
Key facts
- In the models measured, prompt information fades as more tokens are generated, roughly following a power law
- Trained models forget faster (rate ~0.55-0.7) than untrained ones (rate ~0.2-0.5)
- The "conditional no-hair theorem", proved for finite-depth pre-LN softmax transformers under stated conditions, says surviving prompt information must be carried by the generated text
- A simplified linear relay model confirmed the mathematical bound and observed behavior
Sources
If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work
Keep reading
How it worksTurboMamba combines Mamba and Transformer for faster, smarter language modelsHerald of AI
TurboMamba combines Mamba and Transformer for faster, smarter language models
A new hybrid architecture boosts speed and accuracy by mixing two kinds of AI designs.
How it worksAll-transistor transformer learns to do in-context learningHerald of AI
All-transistor transformer learns to do in-context learning
A new physical circuit made entirely of transistors shows that hardware-level AI can perform the same kind of fast, on-the-fly learning as large language models.
How it worksHow Meta boosts engineering productivity by 3× using agents to propagate ad model techniquesHerald of AI
How Meta boosts engineering productivity by 3× using agents to propagate ad model techniques
Meta's new AI agent system adapts small improvements across thousands of ad models automatically, saving millions in computing costs.