Running 284B DeepSeek-V4 on a 64GB CPU with new kernel patches

A new set of kernel patches and quantization tools called llama-modchip allows developers to run the 284-billion parameter DeepSeek-V4-Flash model on a standard 64GB CPU-only machine. Published on November 8, 2026, this project demonstrates that massive Mixture-of-Experts (MoE) architectures can operate efficiently in system RAM without requiring expensive GPU clusters or causing memory overflow errors.

What happened

The primary challenge with large language models has always been memory capacity. Standard builds of DeepSeek-V4-Flash typically require over 76 GiB of storage, which exceeds the usable RAM on most 64GB Linux systems. This mismatch usually forces the operating system to use slow disk swapping or results in out-of-memory crashes. The llama-modchip project solves this by compressing the model checkpoint to 56.56 GiB, allowing the entire model and its context window to fit comfortably within 62.5 GiB of usable Linux RAM.

Beyond just fitting the model into memory, the project introduces significant performance optimizations for CPU inference. By replacing standard matrix-vector loops with multi-column tiling schedules and leveraging AVX-512BW instructions, the patches boost prefill speeds by 77% and decode speeds by 53%. On a 16-core Zen 5 processor, this translates to 44.4 tokens per second during prefill and between 9.1 and 14.7 tokens per second during generation, making interactive use feasible on consumer hardware.

The project also supports smaller MoE models like Nemotron-3.5-30B-A3B, which can run on laptops with 16–32 GB of RAM. In these configurations, the system achieves speeds between 25.0 and 60.4 tokens per second. For users with hybrid setups combining a CPU with a 16 GB GPU, the tool can offload hot attention layers to the GPU while keeping cold routed experts in CPU RAM, pushing throughput to over 80 tokens per second.

How it works

The core innovation lies in a custom quantization method called Fractal-64 and specific kernel patches for the ggml library. Fractal-64 uses a hybrid approach where sensitive layers like embeddings and shared experts are kept in higher precision formats such as Q4_K or F32, while the bulk of the routed MoE experts are compressed to IQ1_S or Q2_0. This reduces the effective bit-width to as low as 1.71 bits per weight for DeepSeek-V4-Flash without significantly degrading model quality.

To manage data movement efficiently, the system employs a Stockfish-PVS v2 drafter, a tiny 1.64 MB n-gram model that predicts likely next tokens. This allows for self-speculative decoding, where the CPU verifies multiple tokens at once. Additionally, the kernel patches use non-temporal streaming hints to ensure that loading cold expert weights from RAM does not evict the hot key-value cache lines from the L3 cache, maintaining high hit rates for active context.

Key details

• DeepSeek-V4-Flash fits into 56.56 GiB of disk space and runs within 57.61 GiB of VmRSS on a 64GB Linux system.
• The project achieves 44.4 tokens per second prefill and 9.1–14.7 tokens per second decode on a pure 64GB CPU setup.
• Nemotron-3.5-30B-A3B runs on 16–32 GB RAM laptops at speeds ranging from 25.0 to 60.4 tokens per second.
• Custom GGUF tensors use type ID 40 (GGML_TYPE_Q2_0), meaning unpatched tools like standard Ollama or LM Studio will reject these models.
• Context restoration for Claude Code prompts with 40 tools takes only 13.1–32.6 milliseconds using /dev/shm KV slot snapshots.
• The default configuration evaluates all 6 routed experts plus one shared expert per layer with zero expert dropping.

Why it matters

For software engineers and technical founders, this development lowers the barrier to entry for deploying state-of-the-art AI models. It eliminates the need for costly GPU infrastructure for many inference tasks, allowing teams to run powerful models on existing server hardware or even high-end developer laptops. This shift can significantly reduce operational costs and simplify deployment pipelines for applications that require long context windows, such as code analysis or document review.

The ability to run 256K context windows entirely in RAM also changes how developers approach agent workflows. With instant context restore times, agents can maintain complex states across multiple tool calls without the latency penalties associated with disk-based storage. This makes local, private AI assistants more viable for enterprise environments where data sovereignty and speed are critical concerns.

What you can do

• Check your CPU readiness for AVX-512 or NEON support using the provided Python script before attempting installation.
• Use the pre-compiled llama-server-modchip-linux-x86_64 binary to avoid compatibility issues with standard llama.cpp builds.
• Start with the Nemotron-30B model if you have less than 64 GB of RAM to test performance on your specific hardware.
• Enable FlashAttention with the -fa on flag when running DeepSeek-V4-Flash to maximize prefill throughput.
• Monitor L3 cache usage and token acceptance rates via the web telemetry HUD to fine-tune your thread count.
• Avoid using standard quantization tools; instead, use the ./modchip quantize command to transcode models into the required Fractal-64 format.