Running a 180B MoE model on 32GB RAM with Transformers
A developer optimized Hugging Face Transformers to run Qwen3.8-Flash-Next on a MacBook M5, achieving near-native speeds by streaming experts from disk.
A software engineer successfully ran the 180-billion parameter Qwen3.8-Flash-Next model on a MacBook M5 with only 32GB of unified memory. By modifying the Hugging Face Transformers library to stream Mixture of Experts (MoE) weights directly from disk, the setup achieved inference speeds comparable to specialized engines like Strata. This experiment demonstrates how consumer hardware can handle massive models through aggressive weight offloading and caching strategies.
What happened
The author, Tarek Ziadé, sought to reduce reliance on cloud APIs by running large language models locally. His primary workstation, a dual-RTX 4090 PC named Orthrus, was often occupied by family gaming activities, prompting him to explore local inference on his work laptop, a MacBook M5 with 32GB of RAM. Initial attempts using standard tools like llama.app and vLLM revealed significant limitations when handling MoE models that exceeded available memory. While some engines could offload weights to CPU RAM, they suffered from performance penalties due to PCIe bandwidth bottlenecks or lack of efficient caching mechanisms for sparse expert activation.
Ziadé focused on optimizing the Hugging Face Transformers library, which traditionally loads entire modules into memory before execution. Standard offloading in Transformers copies all experts in a layer to the GPU for every token, regardless of which ones the router selects. This approach is inefficient for MoE models, where only a small fraction of experts are active per token. To address this, he developed a new streaming mechanism that loads only the required experts from disk into a bounded cache on the device. The resulting implementation, detailed in pull request #50717, allows Transformers to handle models far larger than physical memory by treating disk storage as an extension of VRAM.
How it works
The core innovation involves replacing the standard module loading process with a slot-based caching system. Instead of copying entire expert banks to the GPU, the system maintains a preallocated tensor pool on the device. A lookup table maps specific expert IDs to slots in this pool. When the model’s router selects active experts for a token, the system checks if they are already in the cache. If not, it reads the quantized expert weights from disk via multiple I/O threads. This allows the existing grouped_mm kernel to operate on the cached slots without modification, avoiding the need for custom matrix multiplication implementations.
To further reduce data transfer overhead, the experts are quantized to 3 bits offline, shrinking the 121GB FP8 expert set to 49GB on disk. The dense layers are quantized to 8 bits, occupying roughly 5.4GB. A separate disk path handles the 51GB n-gram embedding table, reading rows only when needed. The system also employs prefetching, where the router predicts experts for subsequent layers while the current layer computes, hiding some of the disk latency. Metal kernels compiled at runtime handle operations lacking fast paths on macOS, such as Gated DeltaNet and hyper-connections.
Key details
- The experiment used Qwen3.8-Flash-Next, a 180B parameter MoE model with 48 layers and 512 experts per layer.
- Inference ran on a MacBook M5 with 32GB unified memory, using approximately 25.4GB at peak during a 6K prompt.
- Experts were quantized to 3 bits, reducing disk footprint from 121GB to 49GB, while dense layers used 8-bit quantization.
- The optimized Transformers implementation achieved 13.9 tokens per second on short decode tasks, compared to 14.8 tok/s for Strata.
- Prefill speed reached 228 tokens per second for a 4K prompt, slightly outperforming Strata’s 220 tok/s.
- The solution requires no C++ extensions, relying on Python, runtime-compiled Metal shaders, and prebuilt kernels from the Hub.
Why it matters
For developers building AI applications, this work highlights the potential of running state-of-the-art models on commodity hardware without expensive GPU clusters. It challenges the assumption that large MoE models require specialized inference engines or vast amounts of VRAM. By leveraging disk streaming and efficient caching, engineers can deploy larger models locally for testing, debugging, or privacy-sensitive tasks. The approach also provides a blueprint for optimizing memory-bound workloads in other domains where data sparsity can be exploited.
However, the results also underscore the limitations of general-purpose frameworks like PyTorch on non-NVIDIA hardware. The remaining performance gap between Transformers and Strata stems largely from host-GPU synchronization issues and the inability to fully overlap computation with I/O at the kernel level. This suggests that while software optimizations can bridge much of the divide, deeper integration with hardware-specific features is necessary for maximum efficiency. Developers should weigh the flexibility of Transformers against the raw performance of dedicated engines when choosing their inference stack.
What you can do
- Install the experimental branch of Transformers from pull request #50717 to test expert streaming on your own hardware.
- Convert MoE model experts to lower-bit quantizations (e.g., 3-bit) to reduce disk I/O requirements and cache footprint.
- Use a high-speed NVMe SSD to minimize latency when streaming weights from disk to device memory.
- Monitor memory usage closely, as the system operates near the limits of available RAM, requiring closed background applications.
- Experiment with prefetching strategies to hide disk read latency during multi-layer computation.
- Review the
moe-expert-streamingSpace for code examples and benchmark details to adapt the approach to other MoE architectures.



