Building with AI

DeepSeek V3 architecture implemented in 190 lines of JAX

A developer recreated the DeepSeek V3 architecture in pure JAX, training a 36M parameter model on a MacBook Air to demonstrate key mechanisms like MLA and MoE.

A glass cube with circuits next to old books
Illustration generated for this article

Vaibhaav Tiwari has released an open-source implementation of the DeepSeek V3 architecture using only 190 lines of pure JAX code. The project, published on GitHub in November 2026, demonstrates that complex modern large language model structures can be understood and executed on consumer hardware without heavy frameworks. By stripping away libraries like PyTorch, Flax, or Equinox, the implementation offers a transparent view of how components like Multi-head Latent Attention and Mixture of Experts function at a fundamental level.

What happened

The core of this release is a single Python file, deepseek_v3.py, which contains the entire model definition. Tiwari trained this model from scratch on a MacBook Air equipped with an M5 chip and 24 GB of memory. The training data consisted of 207 public-domain detective fiction books, totaling approximately 21.7 million tokens. This choice of dataset ensures legal clarity while providing a coherent linguistic structure for the model to learn.

The training process ran for 12,000 steps, processing about 98 million tokens over roughly 2.6 hours. The resulting model contains 36.2 million parameters, with 13.6 million active per token due to its sparse expert routing. Despite its small scale compared to industrial models, the implementation successfully replicates the architectural innovations described in the original DeepSeek-V3 technical report. The project includes not just the code, but also the trained checkpoint, tokenizer, and scripts to reproduce the results or generate new text.

How it works

The implementation relies on plain JAX functions and dictionaries of arrays, avoiding any high-level abstractions. This approach allows engineers to read the code top-to-bottom and understand exactly how data flows through the network. Key architectural features include Multi-head Latent Attention (MLA), which compresses key-value caches to save memory, and DeepSeekMoE, which uses a combination of shared and routed experts to handle computation efficiently.

One notable mechanism is the auxiliary-loss-free balancing strategy. Instead of using a complex loss term to force experts to share the workload equally, the model applies a per-expert bias that nudges routing toward idle experts. This changes which experts are selected for a given token without altering their weights. The code also implements Multi-token Prediction (MTP), where an additional module predicts the token after the next one, helping to stabilize training. All computations use bfloat16 precision on top of float32 master weights, ensuring numerical stability during the update steps.

Key details

  • Codebase size: The core model logic fits in 190 non-blank, non-comment lines of pure JAX.
  • Hardware requirements: Training was performed on a MacBook Air M5 with 24 GB RAM, achieving ~10.7k tokens/s.
  • Model scale: 36.2 million total parameters with 13.6 million active per token via sparse routing.
  • Training data: 21.7 million tokens from 207 public-domain crime and detective novels.
  • Performance: Validation loss dropped from 5.02 to 3.51 over 12,000 steps.
  • Balancing efficacy: Expert load varied between 5.7–6.9% with bias, compared to 0.09–18.6% without it.

Why it matters

For software developers and ML engineers, this project serves as a high-fidelity educational tool. Modern LLM architectures are often obscured by massive codebases and proprietary infrastructure, making it difficult to grasp how specific innovations like MLA or decoupled rotary embeddings actually work. By reducing the implementation to its essential mathematical operations, Tiwari provides a reference that demystifies these concepts. Engineers can inspect the code to see exactly how caching is handled or how routing biases are updated, bridging the gap between theoretical papers and practical implementation.

Furthermore, the ability to train a representative architecture on a laptop challenges the assumption that meaningful experimentation requires cluster-scale resources. While the resulting model is not competitive with state-of-the-art systems, it proves that the structural integrity of advanced designs can be validated on consumer hardware. This lowers the barrier to entry for researchers and students who want to experiment with architectural modifications, such as changing the number of experts or adjusting attention mechanisms, without needing access to expensive cloud compute.

What you can do

  • Clone the repository and install the pinned dependencies using uv to ensure compatibility with jax-metal.
  • Run the provided tests to verify your environment supports the necessary JAX operations on Apple Silicon.
  • Execute the data preparation script to download and tokenize the public-domain corpus, or download the pre-processed binaries for exact reproducibility.
  • Train the model locally using the default configuration or the smaller config for machines with less memory.
  • Generate text samples using the provided checkpoint to observe the model's output on detective fiction prompts.
  • Study the deepseek_v3.py file line-by-line to understand the implementation of MLA, MoE routing, and multi-token prediction.
  • #jax
  • #deepseek
  • #machine learning
  • #open source
  • #llm architecture

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories