How Language Models Slowly Forget the Start of Their Prompts

Forgetting Is Built In

A new research paper shows that language models naturally lose track of their early prompt instructions over time. The finding, published on arXiv by Shuyang Xiang, gives a name to a behavior researchers have long observed: the farther a model gets into generating text, the less it carries information from the very beginning of its prompt.

The paper calls this the “conditional no-hair theorem,” borrowing a term from physics. In astrophysics, the no-hair theorem says black holes can be described by only a few properties, losing all other details about what fell in. Here, the idea is similar: given the text a model has already generated, the original prompt’s influence fades away.

Measuring How Fast Models Forget

The researchers compared two different prompts that were followed by the exact same continuation text. Since the later output was identical, any difference in the model’s internal states had to come from the prompt alone. They measured how that difference shrank as more tokens were generated.

They found the gap between the two internal states decays roughly as a power law — meaning it shrinks like one divided by t to some power, where t is the number of tokens generated. A higher power means faster forgetting ¹.

Trained models, like the ones used in everyday AI applications, forget relatively quickly, with a decay rate around 0.55 to 0.7. Untrained models start out with a much slower rate, around 0.2 to 0.5, meaning they hold on to prompt information longer. The researchers also proved a mathematical bound showing exactly how fast this decay must happen, depending on prompt length and model depth.

Why It Happens

The mechanism is geometric. As the context window fills up, the direct attention the model pays to the original prompt gets diluted across an ever-growing amount of text. Later tokens act as relay runners, passing along whatever they picked up from earlier tokens — but this relay can only slow the forgetting by a small logarithmic amount, not stop it entirely.

To test this idea, the team built a simplified linear model that can be solved exactly. It reproduced the same slowing decay seen in real untrained networks and confirmed the mathematical bound cannot be improved.

When the researchers added a logit bias to cancel out the dilution effect, they found forgetting slowed down early in training and actually reversed after further training. This suggests that dilution is both necessary and sufficient for the forgetting they measured ¹.

What Survives Moves to the Text

The key implication is that any property of the prompt that “survives” — such as the language it was written in — cannot stay encoded in the model’s internal states. It has to be carried forward by the generated text itself. This is why long-form generation often drifts: the model is continually rewriting its own memory into the output.

The paper adds a rigorous theoretical foundation to something practitioners have noticed anecdotally: the further a model writes, the more it relies on what it has already produced rather than on its original instructions.

This story reports on a trend for information only. It is not advice or encouragement to do any of it.

Key facts

• Language models lose prompt information as they generate more tokens, following a predictable power-law decay
• Trained models forget faster (rate ~0.55-0.7) than untrained ones (rate ~0.2-0.5)
• The "conditional no-hair theorem" shows any surviving prompt info must be carried by generated text, not internal states
• A simplified linear relay model confirmed the mathematical bound and observed behavior

Sources

1. A Conditional No-Hair Theorem for Transformers: How Language Models Forget Their Prompt