Stanford gives frozen AI models a super-powered memory that needs only 16 tokens
Researchers turned a whole library of documents into 16 tiny signals so a locked-down AI can answer questions without rethinking everything each time.
What you could do with it
- A company could give its locked-down AI access to a massive library of documents without slowing down or increasing costs every time a question comes in 1.
- A researcher could train a memory add-on on thousands of pages once and then reuse it with a frozen language model so new questions never require re-processing the whole collection 1.
- A business could add newer documents to an existing store one at a time without being forced to re-encode documents already stored there 1.
How it works: a shortcut between question and library
A language model normally answers by reading whatever you paste in front of it — its "context window." Pasting millions of words grows slower as the pile gets bigger. Another option, called Retrieval-Augmented Generation (RAG), finds the most relevant chunks and pastes only those, but it can miss subtle clues hidden elsewhere in the collection.
Memorilla takes a different path. Think of it like a librarian who has already read every book in a vast branch and then holds just a single index card summarizing what she knows about any particular topic. That index card is made up of 16 learned "tokens" — tiny numerical signals — produced whenever a new question arrives.
The system runs in three steps. First, a frozen embedding model called Qwen3-Embedding-4B turns every document in the collection into numbers one time; adding later documents does not force older ones to be re-encoded. Second, when a question comes in, 16 learned memory slots scan the whole store through cross-attention layers and pull out exactly what seems useful. Third, those slots are projected into the decoder's input space and placed in front of the question so the frozen Qwen3-8B model can read them along with the prompt itself.
Only the memory module trains. It carries 194 million parameters, while both the encoder and decoder stay completely frozen throughout. Training happens in three stages: Wikipedia paraphrase reconstruction, a mix of ten question-answering and summarization datasets, and then fine-tuning across eight final benchmarks. The full recipe runs about 19 hours on four H100 graphics cards 12.
Benchmarks: faster and sharper on average, but not perfect
Across eight standard tests, Memorilla posted the highest reported average score of 49.68, beating other published baselines such as xRAG at 41.92, LLMLingua-2 at 40.75 and Mem0 at 38.02 1. It led on five of the eight measures, including a PersonalizationV4 accuracy of 44.60 percent compared with 27.92 percent for plain full-context prompting, and top ROUGE-L scores on LaMP-4 and LaMP-7 .
On FactKG, a factual question-answering test, Memorilla reached 84.33 percent accuracy with only 16 tokens, versus 80.43 percent when every document was pasted directly — using 108 times fewer document tokens 1. Compression peaked on NarrativeQA, where those 16 tokens replaced 67,385 document tokens, a 4,212-fold reduction . Even as agent memory, cutting down to eight tokens lifted a Qwen3-4B TextWorld agent from 18.6 percent to 24.2 percent average success .
It is not flawless. On PubMedQA, which contains only 324-token medical abstracts, Memorilla scored 59.00 percent while LLMLingua-2 hit 69.00 percent, because there is almost nothing to compress 1. MemoRAG also outranked it on PersonalizationV4 and NarrativeQA ROUGE-L, though MemoRAG used far more document tokens in those cases .
Speed and how to run it yourself
On a single H100 GPU, answering with 512 documents — about 128,000 tokens — took Memorilla 0.74 seconds, while feeding the same documents directly slowed response from 0.75 seconds to 19.9 seconds, 27 times slower; the memory module itself added only about two milliseconds 1. An optional precomputed cache brings a question down to 8.1 milliseconds even at one million documents on a B200 GPU, though holding that cache at four million documents requires 80 gigabytes of memory .
The project ships under an Apache 2.0 license with code, training recipes and checkpoint folders publicly available on Hugging Face under the name memorilla/Memorilla-Qwen3-8B, along with a separate PersonalizationV4 dataset covering 149 synthetic users 12. Running it requires Linux, CUDA-capable GPUs, Python 3.11 and at least one 80-gigabyte GPU for quick starts .
This story reports on a trend for information only. It is not advice or encouragement to do any of it.
Key facts
- Memorilla compresses thousands of document tokens into just 16 fixed-size signals for a frozen language model
- The 194-million-parameter memory module stays lightweight while encoder and decoder remain frozen
- It averaged 49.68 across eight benchmarks on frozen Qwen3-8B, leading all tested RAG-style baselines
- On H100 hardware, it answered from 512 documents in 0.74 seconds — 27 times faster than full-context at that size
- Code, checkpoints and training recipes are public under Apache 2.0 on Hugging Face
Sources
If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work
Keep reading
Researchers develop method to make large language models respond faster
A new training and inference approach called DScout could help AI systems generate text up to four times faster while maintaining accuracy.
How it worksDeepSeek V3 architecture re-implemented in 190 lines of code, trained on a MacBook AirHerald of AI
DeepSeek V3 architecture re-implemented in 190 lines of code, trained on a MacBook Air
A developer showed how to fit DeepSeek V3's architecture into a tiny codebase — good for learning, not ready to chat.
How it worksHow Echo lets big AI models train faster using ultra-low 4-bit dataHerald of AI
How Echo lets big AI models train faster using ultra-low 4-bit data
A new communication method cuts the data-hauling bottleneck in large AI training by sending just 4 bits at a time, without breaking model quality.