ThinkBit Two-bit compression keeps AI reasoning sharp while cutting inference costs 8×

Reasoning AI models can spit out thousands of words (tokens) for a single answer. That makes them slow and expensive because they have to move large amounts of data around. Storing the model’s weights— the numbers that control how it thinks— in two bits instead of the usual sixteen-bit bfloat16 format could make inference eight times cheaper, according to the paper published on 26 October.

The catch is that most two-bit techniques are tested on easy multiple-choice questions or perplexity scores, which measure how well a model predicts the next word. Those tests say little about whether the model can still work through tough math or logic puzzles. ThinkBit was built specifically to preserve that reasoning power.

What you could do with it

• Researchers could compress reasoning language models into two-bit storage, potentially cutting inference costs roughly eightfold while keeping the model's ability to solve hard math and logic puzzles intact ¹.
• AI app developers could run reasoning-capable models faster and cheaper on everyday devices like phones, since storing weights in two bits instead of sixteen-bit format reduces memory demands dramatically ¹.
• A researcher could evaluate a compressed model on the MATH-500 benchmark to confirm that the quantization process did not destroy its reasoning power ¹.

A three-step recipe for sharp reasoning

ThinkBit treats every weight as a tiny integer codeword multiplied by a floating-point scale that is shared by a group of nearby weights. The framework first assigns each layer of the network a bit-tier— either 2, 3, or a ternary option— based on how sensitive that layer is to compression. This means important layers get a bit more precision.

Next, the system searches for the best codewords block by block using a method called Gumbel-softmax. Because looking at each layer in isolation can miss how they interact, the team then recalibrates the scales while keeping the codewords fixed. This step uses knowledge distillation: the compressed model is taught to mimic the teacher model’s output distributions.

Finally, a short quantization-aware training (QAT) stage lets the model reassign codewords to better match the original distribution. To pick the final checkpoint, the researchers perform a paired reasoning test on problems from the training set and choose whichever version— the recalibrated one or the QAT one— performs better.

Numbers that matter

When tested on the 1.7-billion-parameter Qwen3-1.7B model, the recalibrated checkpoint scored 46.7 on the MATH-500 benchmark (a set of 500 math problems), compared with 30.0 for another method called EfficientQAT trained on the same data. After the QAT step, the score jumped to 73.0.

At just 0.52 GB, ThinkBit’s compressed model is smaller than any published two-bit GGUF version of Qwen3-1.7B and still more accurate on MATH-500. The approach scales up: for a 14-billion-parameter model, a 3.74 GB checkpoint scored 94.8 on MATH-500, against 97.5 for the full 29.5 GB bfloat16 version.

What comes next

The researchers show that ThinkBit can preserve reasoning while drastically cutting memory usage. If the method is adopted, it could make reasoning-capable AI cheaper to run on laptops, phones, and cloud servers. The paper is available on Hugging Face, and the authors are from Microsoft.

This story reports on a trend for information only. It is not advice or encouragement to do any of it.

How good is it

On the 1.7-billion-parameter Qwen3 model, ThinkBit's recalibrated checkpoint scored 46.7 on the MATH-500 benchmark, compared with 30.0 for EfficientQAT trained on the same data ¹. After a short quantization-aware training stage, the score rose to 73.0, and the model weighed just 0.52 GB, making it smaller than any published two-bit GGUF version of Qwen3-1.7B while still more accurate . The approach also scales up: on a 14-billion-parameter model, a 3.74 GB checkpoint scored 94.8 against 97.5 for the full 29.5 GB bfloat16 model, showing only a small drop in reasoning quality for a fraction of the size . The MATH-500 benchmark measures performance on 500 math problems, so higher scores indicate better problem-solving ability .

Key facts

• ThinkBit shrinks AI models to two‑bit storage, cutting inference costs by up to 8×.
• It preserves reasoning ability, scoring 73.0 on MATH‑500 for Qwen3‑1.7B after quantization‑aware training.
• The compressed model is 0.52 GB— smaller than any other two‑bit GGUF version of the same model.
• For a 14B‑parameter model, a 3.74 GB checkpoint scored 94.8 on MATH‑500, near the full‑precision score of 97.5.
• The method uses layer‑wise bit allocation, Gumbel‑softmax codeword search, and knowledge‑distillation recalibration.

Sources

1. Paper page - ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning