PeekKV A smarter way to compress AI memory that boosts long-context speed
Researchers found that looking at just one token can decide which parts of a large language model's memory matter, cutting costs without losing accuracy.
The memory problem with long AI conversations
As Llama-3.1 and other large language models handle longer and longer inputs, they face a growing bottleneck: the KV cache. This cache stores information the model needs to remember from earlier in a conversation, but its size grows linearly with context length, making it expensive for services running these models.1
This has led researchers to look for ways to compress the KV cache. Instead of keeping everything equally important, some methods try to keep only the most useful parts and throw away the rest.
Why existing methods struggle
The researchers behind the new technique, Minyoung Go and Yongjun Park, discovered a key reason why current approaches are inconsistent. They found that the attention heads — the parts of the model that decide which information to focus on — shift between two stages of how a language model processes text.
The first stage, called prefill, reads the entire input at once. The second stage, called decode, generates the response token by token. Surprisingly, even within a single input, different heads become active between these two stages.1
However, they also found something important about the decode stage. Once they filtered out format tokens — bits of text that are just structure like bullet points or section headers rather than real content — the set of active heads stayed remarkably stable across all content tokens in the response.1
The one-token trick
This insight led the researchers to develop PeekKV, a method that requires no offline profiling or training. Instead, it works on any model or workload right away.
The core idea is simple but powerful: after the prefill stage finishes, PeekKV looks at just a single content token. This one-token peek is enough to figure out which head groups are most important and which tokens within each group carry the most weight.1
Using these scores, the system allocates its limited KV cache budget across head groups in proportion to their importance, and within each group keeps only the top-scoring tokens.1
This means PeekKV adapts to whatever input it receives, rather than relying on fixed assumptions about which parts of the model will always matter.
Results across benchmarks
The researchers tested PeekKV across four public benchmarks. It scored the best accuracy in over half of the per-model comparisons against state-of-the-art compression methods.
On Llama-3.1-8B-Instruct specifically, PeekKV retained 95.7% of full-cache accuracy while using only 10% of the memory on RULER, a benchmark for reasoning over long contexts of 16,000 tokens. On LongBench, it kept 95.9% of full accuracy with just 1.5% of the memory budget.1
The paper was submitted to arXiv in October 2026 by researchers Minyoung Go and Yongjun Park.
This story reports on a trend for information only. It is not advice or encouragement to do any of it.
Key facts
- PeekKV uses just one token after the prefill stage to determine which memory components matter most
- On Llama-3.1-8B-Instruct, it kept 95.7% accuracy at 10% memory budget on the RULER benchmark
- It retained 95.9% accuracy with only 1.5% memory on LongBench
- The method requires no offline profiling or training and works on unseen workloads immediately
- Attention heads become active differently between prefill and decode stages within a single input