DeepSeek V3 architecture re-implemented in 190 lines of code, trained on a MacBook Air
A developer showed how to fit DeepSeek V3's architecture into a tiny codebase — good for learning, not ready to chat.

A tiny teaching copy of a large language model, written in 190 lines and trained on a laptop.
- Download the code from GitHub
- Run a script to grab detective-fiction books
- Train the model on your Mac for a few hours
- Type a prompt and see text appear
What you could do with it
- A student could read the 190-line codebase top to bottom to see how DeepSeek-V3's key design ideas work inside their own implementation 1.
- A hobbyist with an Apple Silicon Mac could reproduce the training run themselves, building the dataset and running the model from scratch 3.
- A developer could study how techniques like Multi-head Latent Attention and auxiliary-loss-free load balancing are implemented in practice 1.

Picture: vaibhaav.in
Words to know
- Multi-head Latent Attention (MLA): stores one small number summary per token instead of full details, saving memory · Mixture of Experts (MoE): picks only a few specialized sub-networks per input instead of running all of them · auxiliary-loss-free balancing: a trick that nudges experts to share work evenly without adding a separate cost term to training
This teaching demo vs. the real DeepSeek‑V3
Figures from Vaibhaav Tiwari’s blog post and GitHub readme; DeepSeek‑V3 specs from its December 2024 technical report.
What the project is
Tiwari's GitHub repo contains a self-contained re-implementation of the key ideas behind DeepSeek-V3, the large open model released in December 2024 by DeepSeek-AI 12. The file deepseek_v3.py is 190 lines of plain Python/JAX: there are no extra libraries such as Flax or Equinox, every parameter is a standard dictionary of arrays, and each piece of the model is a single function you can read top to bottom . The accompanying blog post at vaibhaav.in explains the code step by step and shows diagrams 3.
He describes the result as "an architecture demo, not a strong language model" 1.
How it was trained
The small model has 36.2 million parameters total, with 13.6 million active for each token because it uses a Mixture-of-Experts design 1. It was pre-trained from scratch for about 12,000 steps (roughly 98 million tokens, or 4.5 passes over the data) using Detective-fiction books from Project Gutenberg, mostly by Dashiell Hammett, which were included three times . Validation used held-out crime fiction . Training ran on a MacBook Air with an M5 chip and 24 GB of unified memory through JAX's Metal plugin, which lets Apple Silicon GPUs be used as accelerators for JAX 3. The run finished in about 2.6 hours at a median speed of roughly 10,700 tokens per second and reached a final validation loss of 3.51 . For comparison, validation loss dropped from 5.02 at step 250 to 3.51 by the end .
The dataset was downloaded automatically by a script called prepare_data.py, which cleans the books, builds an 8,192-token Byte-Pair-Encoding tokenizer, and writes train.bin and val.bin files 1.
Key techniques shown
The implementation includes four features that come straight out of the DeepSeek-V3 paper 12:
- Multi-head Latent Attention (MLA): each layer caches only 144 numbers per token instead of 480, combining shared keys with compressed values 1.
- Decoupled rotary position encoding (RoPE): one rotary key is kept outside the compressed latent and applied separately 1.
- DeepSeek-MoE: one always-on shared expert plus 16 routed experts, with each token sent to the top 4 1.
- Auxiliary-loss-free load balancing: a small per-expert bias shifts routing toward idle experts; this made expert usage range from 5.7% to 6.9% per expert instead of the very uneven 0.09% to 18.6% seen without the bias, while validation loss stayed the same 1.
- Multi-token prediction (MTP): an extra head also predicts the next-next token, weighted at 0.3 in the loss 1.
Compared with the full DeepSeek-V3 model described in the December 2024 technical report, this is a teaching-scale version focused on showing structure, not matching production quality 12.
How others can try it
The release includes a checkpoint (ckpt.pkl, 138 MB) and a tokenizer 1. To reproduce the run, users need an Apple Silicon Mac with Python 3.12 and the package manager uv; JAX must stay at version 0.4.34 because the metal backend is pinned to jax-metal==0.1.1 . CPU-only training is mentioned but longer . A short note warns that the checkpoint is a Python pickle file and should only be loaded from trusted sources . Sample generation with the prompt beginning "The fog came in off the..." produces output like "The fog came in off the bay, and the great trees were spreading and the stars were shining" .
Full instructions, code, plots and the detailed write-up are available on GitHub under Vaibhaav-Tiwari/deepseek-v3-jax 1.
What this doesn't do
The project does not cover the full DeepSeek-V3 architecture at its published scale, which contains hundreds of billions of parameters and requires clusters of data-center GPUs 2. It stops at a ~36-million-parameter model trained on about 21.7 million tokens of public-domain detective fiction 1. No benchmarks against other language models were reported .
How good is it
The project itself acknowledges the result is not strong: it describes the output as 'an architecture demo, not a strong language model' 1. It reached a final validation loss of 3.51 after 12,000 steps on ~98 million tokens of detective-fiction books . Validation loss dropped from 5.02 at step 250 to 3.51 by the end, and there are no rival benchmarks reported .
Corrections and updates
- Corrected on 15 November 2026. An earlier version said the model was trained on less than 25 million words; the source states 21.7 million training tokens, measured in tokens rather than words, and specifies the genre more precisely as detective fiction.
Key facts
- 190 lines of pure JAX implement DeepSeek-V3-style MLA, MoE and multi-token prediction
- Trained on a MacBook Air M5 (24 GB) in ~2.6 hours, reaching validation loss 3.51
- Model has 36.2M parameters, 13.6M active per token, learned from 21.7M detective-fiction tokens
- Per-expert load-balancing bias made expert usage even (5.7–6.9%) without changing loss
- Released as an architecture demo on GitHub with checkpoint and training scripts
Sources
If this story concerns you or your company, or uses something that belongs to you, ask for a correction or for it to be taken down. How requests work
Keep reading
Business and rulesMirendil raises $1 billion to use AI agents for faster model developmentHerald of AI
Mirendil raises $1 billion to use AI agents for faster model development
A new startup founded by former Google DeepMind and Anthropic researchers says its automated platform lets small teams build large AI models quickly and cheaply.
Stanford gives frozen AI models a super-powered memory that needs only 16 tokens
Researchers turned a whole library of documents into 16 tiny signals so a locked-down AI can answer questions without rethinking everything each time.
How it worksHow Meta boosts engineering productivity by 3× using agents to propagate ad model techniquesHerald of AI
How Meta boosts engineering productivity by 3× using agents to propagate ad model techniques
Meta's new AI agent system adapts small improvements across thousands of ad models automatically, saving millions in computing costs.