Specializing the Vocabulary of Small Language Models Cuts Inference Cost by 77%

Leaving the Tokenizer Behind

For most developers who fine-tune a general-purpose Qwen model for a specific task, the tokenizer is inherited and forgotten. It is set once before any training begins and then treated as fixed.

A team at Dharma-AI argues that this leaves money on the table. In an automated grader for Portuguese essays running on Qwen3-4B, the researchers rebuilt the vocabulary using a method they call VocabX and cut measured inference cost by 77 percent without changing the base model or the training data ¹. The result comes from a pre-publication paper that may shift before peer review.

The Two Kinds of Waste

General-purpose tokenizers are trained on huge corpora in many languages and domains, so they split text into tokens by frequency, not by cost to the user ¹. That choice creates two problems for a narrow application.

First, fragmentation. Sequences that repeat on every request, like a JSON wrapper or standard evaluation phrases, are split across many tokens when one token would do ¹. Second, excess. The vocabulary keeps tens of thousands of entries the task almost never uses, yet the model scores all of them at every generated token .

A model graded with Portuguese texts still computes scores for every Chinese token in its 151,669-entry vocabulary at every step of generation ¹.

Reading Is Cheap, Writing Costs More

The savings come from understanding that generating a token is much more expensive than reading one ¹. During the prefill phase, the model processes the full input in a single pass and reuses weight reads across every token. During decode, it produces one token at a time, and each step requires a full pass through the network plus a read of the KV cache, the attention memory that grows with context .

GPU compute moves from being the bottleneck to memory bandwidth. In the grader test, a generated token cost 7.4 times a token read, though the ratio varies with hardware, batch size and context length ¹. Reducing the number of output tokens matters far more than reducing the number of input tokens.

A second cost factor is the vocabulary itself. The model’s final layer, the LM head, holds one row of parameters per entry in the vocabulary, accounting for nearly 10 percent of parameters in a small model like Qwen3-4B ¹. Every generated token forces the GPU to read and multiply the entire matrix, so unused entries cost the same as frequent ones .

VocabX accounts for this asymmetry by selecting vocabulary entries around the cost of each token rather than raw frequency ¹.

What Changed and What Did Not

The grader receives an essay, a prompt and supporting texts, then returns a JSON assessment with a grade and comment for each competency ¹. Before the overhaul, it already used a 4-billion-parameter model fine-tuned for the task; only the tokenizer was replaced .

The numbers are drawn from the current version of the VocabX paper, still under review, and the authors note they may change before publication ¹.

Key facts

• Inference cost fell by 77 percent on a production Portuguese essay grader after rebuilding the vocabulary with VocabX.
• The base model, training data and training recipe were unchanged; only the tokenizer was replaced.
• Generating a token in the test cost 7.4 times a token read because the decode phase is memory-bandwidth bound.
• The LM head holds roughly 10 percent of parameters in a 4-billion-parameter model and is read in full at every generated token.

The findings are based on the latest version of the manuscript, which is currently undergoing review, and could be modified prior to publication.

Sources

1. How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications