Running 16B LLMs on 1 GB RAM with custom llama.cpp patches
The smolmaxxing project demonstrates how to run large mixture-of-experts models on devices with only 1 GB of usable RAM by streaming weights from flash storage.
A new open-source project called smolmaxxing proves that modern large language models can run on hardware previously considered obsolete. By applying specialized patches to the llama.cpp inference engine, developers have successfully executed 16-billion-parameter models on a Xiaomi TV box and a 2013 iPad Air. The project, published on GitHub in November 2026, relies on aggressive memory management and custom kernels to bypass standard hardware limitations.
What happened
The smolmaxxing repository documents experiments running large mixture-of-experts (MoE) models on devices with severely constrained random-access memory (RAM). The primary test subjects were a Xiaomi TV Box S (3rd Gen) with approximately 1 GB of usable RAM and an Apple iPad Air (2013) with roughly 700 MB of usable RAM. Despite these tight constraints, the project achieved functional inference speeds for models far larger than the available memory.
On the Android-based TV box, the system ran the Ling-mini-2.0 16B model at 5.1 tokens per second. The older iOS tablet managed the same model at speeds between 2.6 and 4.5 tokens per second. For lighter daily tasks, the TV box handled Qwen3.5-2B at 7.7 tokens per second, while the iPad ran a 1-bit quantized Bonsai-1.7B model at 24 to 26 tokens per second. Every performance metric was measured directly on the physical devices, not simulated.
The project emphasizes that these results are not isolated anomalies but the product of reproducible engineering techniques. The authors note that the methods apply to any Linux or Android device where the model size exceeds available RAM, provided the storage medium is fast enough to stream data. Future expansions aim to include smartphones, single-board computers, and routers.
How it works
The core innovation lies in a set of patches applied to llama.cpp, a popular C++ library for running LLMs locally. Standard inference engines typically require the entire model, or at least all active parameters, to reside in RAM. Smolmaxxing breaks this requirement by treating flash storage as an extension of memory. Since MoE models only activate a small subset of "experts" for each token, the system keeps frequently used experts in RAM and streams the rest from flash storage on demand. It also prefetches the next layer’s experts to minimize latency.
To make this streaming viable on weak CPUs, the project introduces custom kernels tailored to specific chip architectures. For the Cortex-A55 processor in the TV box, engineers implemented a 1-bit lookup-table kernel and repacked 2-bit and ternary formats. The Apple A7 chip in the iPad lacks dot-product instructions, so the team developed alternative versions of these kernels that rely only on basic NEON instructions. These optimizations reduce the computational load per operation, allowing slower processors to keep up with the data streaming from storage.
Memory savings are further achieved by pruning the model’s vocabulary. Since most words in a standard vocabulary are rarely used during inference, the project drops unused output rows to free up RAM. On the TV box, a "headless" mode disables the Android launcher and background Google services during inference, ensuring that every available megabyte of RAM is dedicated to the model. This combination of selective loading, custom low-level code, and system-level resource stripping enables 8B and 16B models to operate within a 1 GB memory budget.
Key details
- Hardware tested: Xiaomi TV Box S (3rd Gen) with Amlogic S7D chip and iPad Air (2013) with Apple A7 chip.
- Memory constraints: Both devices operated with approximately 1 GB or less of usable RAM.
- Peak performance: The 16B Ling-mini-2.0 model ran at 5.1 tok/s on the TV box and up to 4.5 tok/s on the iPad.
- Core technique: Mixture-of-experts models are paged from flash storage, keeping only active experts in RAM.
- Software basis: All modifications are patches applied to the llama.cpp inference engine; stock llama.cpp cannot load these models.
- Availability: Ready-made model files are hosted on Hugging Face, with setup guides for ARMv8.2 Android and jailbroken iOS 12 devices.
Why it matters
For software engineers and AI practitioners, this project challenges the assumption that local LLM inference requires high-end GPUs or substantial RAM. It demonstrates that careful systems programming can unlock significant capability from idle consumer electronics. TV boxes, old tablets, and routers often sit unused, yet they possess CPUs and flash storage capable of handling meaningful AI workloads if the software stack is optimized correctly. This opens avenues for deploying edge AI in environments where power, cost, or space prohibit traditional servers.
The techniques also highlight the importance of architecture-specific optimization. General-purpose quantization is often insufficient for extreme edge cases. By writing custom kernels that leverage specific instruction sets like NEON, developers can squeeze out performance that generic libraries miss. This approach is particularly relevant for mixture-of-experts models, which are inherently suited to paging strategies due to their sparse activation patterns. As models grow larger, efficient memory management will become as critical as raw compute power.
What you can do
- Download the patched llama.cpp binaries and ready-made model files from the smolmaxxing Hugging Face repository.
- Test the setup on an ARMv8.2 Android device using the guide in
docs/tv-box-setup.md. - Experiment with a jailbroken iPad on iOS 12.5.7 using the scripts provided in the
ipad-air/directory. - Review
docs/how-it-works.mdto understand the specific gains from each kernel and memory optimization technique. - Rebuild models for your own hardware using the tools in the
tools/directory, following the instructions indocs/models.md. - Read
EXPERIMENT.mdto analyze the 160 experiments and failures documented by the authors, helping you avoid common pitfalls.



