Cutting local LLM swap times from ten seconds to under two

Running multiple large language models on a single GPU often requires swapping them in and out of memory, a process that traditionally incurs significant latency. A new open-source tool named llama-sleep, combined with specific Linux kernel configurations, has reduced this swap time from over ten seconds to under two seconds while preserving cached prompts.

What happened

Developers who self-host local AI models frequently face a bottleneck when their hardware cannot fit all required models into GPU memory simultaneously. In a typical setup using a proxy like llama-swap, switching between two GPU-filling models involves stopping the current process, freeing its memory, and loading the new model from scratch. In baseline tests conducted on an NVIDIA RTX 3060, this cold swap took approximately 10.3 seconds. This delay not only impacts responsiveness but also destroys the key-value (KV) cache, forcing the system to recompute long system prompts upon every switch.

To address this inefficiency, Robert Zaage developed llama-sleep, a lightweight wrapper for llama.cpp’s server. Instead of terminating the model process during idle periods, llama-sleep uses NVIDIA’s CUDA checkpointing features to suspend the model’s state. It copies the GPU memory contents into host RAM, effectively "parking" the model. When the model is needed again, the state is restored directly from RAM to the GPU. This approach eliminates the need to reload model weights from disk and retains the KV cache, allowing the system to resume inference almost immediately.

The initial implementation of this parking mechanism reduced swap times to 2.1 seconds. However, further optimization revealed that the data transfer speed was limited by how the operating system handles memory pages. By enabling multi-size transparent huge pages (mTHP) in the Linux kernel, the developer reduced the swap time even further to 1.7 seconds. This represents a six-fold improvement over the original baseline, making rapid model switching viable for real-time applications like voice assistants and automated workflows.

How it works

The core technology behind llama-sleep relies on the cuda-checkpoint capability introduced in NVIDIA driver version 550. This feature allows a running CUDA process to have its device memory snapshot and moved to host memory without terminating the process. llama-sleep acts as a reverse proxy and process manager. It runs the llama-server as a detached daemon, shielding it from direct external requests. When the model becomes idle for a configured period, such as five seconds, llama-sleep triggers the checkpoint function. This suspends the CUDA context, copies the VRAM data to system RAM, and frees the GPU resources for other tasks.

A critical challenge in this design is handling incoming requests while a model is parked or being restored. If a request arrives while the GPU memory is being copied back, the process could crash or return errors. llama-sleep solves this by holding all incoming requests, including health checks from the proxy, in a queue. It only releases these requests once the model is fully awake and responsive. This ensures that the proxy never sees a "ready" status for a model that is still in transit, preventing segmentation faults and ensuring seamless operation.

The final performance gain comes from optimizing the memory copy itself. The transfer of gigabytes of data between GPU and host RAM is heavily influenced by page faults. Standard 4 KB memory pages require the operating system to manage millions of individual page entries during the copy, creating significant overhead. By enabling multi-size transparent huge pages in Linux kernels version 6.8 and later, the system can use larger memory blocks ranging from 16 KB to 1 MB. This reduces the number of page faults dramatically, allowing the data transfer to approach the theoretical bandwidth limits of the PCIe bus rather than being bottlenecked by CPU memory management.

Key details

• Performance improvement: Swap time between two 26B parameter models dropped from 10.3 seconds to 1.7 seconds.
• Cache preservation: The KV cache remains intact during swaps, eliminating the need to recompute prompts of thousands of tokens.
• Hardware requirements: Requires an NVIDIA GPU with driver version 570 or later and a Linux host system.
• Software dependencies: Uses llama-swap v263, llama.cpp b11552, and the standalone llama-sleep binary written in Go.
• Kernel configuration: Optimal performance requires enabling multi-size transparent huge pages (mTHP) for anonymous memory in the Linux kernel.
• Memory trade-off: Parked models consume host RAM equivalent to their GPU memory usage, requiring sufficient system memory.

Why it matters

For engineers building local AI infrastructure, this development transforms the feasibility of multi-model architectures on consumer hardware. Previously, the high latency of swapping models made it impractical to use specialized models for different tasks, such as a small model for voice commands and a larger one for complex reasoning. Users had to choose between keeping a single large model loaded, wasting resources on simple tasks, or accepting long delays when switching contexts. With swap times under two seconds, systems can dynamically allocate resources based on demand without perceptible lag.

Preserving the KV cache is equally significant for user experience and computational efficiency. Recomputing long system prompts or conversation histories consumes both time and energy. By retaining this state in host RAM, llama-sleep ensures that the first token generation after a swap is nearly instantaneous. This is particularly valuable for applications like home automation or interactive agents where context continuity is essential. The ability to park models in the background means that the system is always ready to respond, bridging the gap between the flexibility of cloud APIs and the privacy of local inference.

What you can do

• Update your drivers: Ensure your NVIDIA GPU drivers are updated to version 570 or later to support CUDA checkpointing features.
• Install llama-sleep: Download the standalone Go binary for llama-sleep and integrate it with your existing llama-swap configuration.
• Configure idle timeouts: Set the --idle parameter in llama-sleep to match your usage patterns, ensuring models park only when not in active use.
• Enable huge pages: If running Linux kernel 6.8 or newer, enable multi-size transparent huge pages by writing "always" to the relevant sysfs entries for sizes between 16 KB and 1 MB.
• Monitor memory usage: Verify that your host system has enough free RAM to hold the parked models alongside other running processes.
• Test swap latency: Measure the time from request to first token after a swap to validate the performance gains in your specific hardware environment.