Open & local AI

Boost llama.cpp inference speed with a single Linux environment variable

Setting GLIBC_TUNABLES to enable huge pages reduces page faults in llama.cpp, yielding up to 21% higher throughput on H100 GPUs.

Illustration comparing small 4KB memory pages to large 2MB huge pages on a circuit board
Illustration generated for this article

A simple configuration change involving Linux memory management has been shown to significantly accelerate local large language model serving. By setting a single environment variable, engineers using llama.cpp and Ollama can reduce CPU overhead caused by memory allocation, resulting in faster response times and higher request throughput without modifying any source code.

What happened

Engineers at Tensward, an open-source tooling company focused on LLM inference optimization, discovered that their llama.cpp server was spending considerable time waiting for the Linux kernel to allocate memory. The bottleneck was not on the GPU, but in how the operating system handled RAM requests for storing conversation snapshots. By enabling "huge pages" through a glibc tunable, they reduced the number of page faults per request from approximately 145,000 to just 1,850 on an NVIDIA H100 GPU.

The team tested this adjustment on Qwen3.8-27B, a widely used model in local AI deployments. The change required no code recompilation or root privileges, only the addition of GLIBC_TUNABLES=glibc.malloc.hugetlb=1 before starting the server process. This adjustment instructs the memory allocator to request larger contiguous blocks of memory, minimizing the kernel's intervention during write operations.

Performance gains varied by hardware and workload. On an H100 handling tool-calling prompts, throughput increased by 21%, and the median time to the first token dropped by 22%. Even on slower hardware like the NVIDIA L4, plain chat scenarios saw the first token arrive 22% sooner, although overall throughput improvements were more modest at 3.5%. These results suggest that the benefit is most pronounced when the CPU memory allocation latency becomes a significant fraction of the total request time.

How it works

Linux typically allocates memory in small 4 KB chunks called pages. When a program like llama.cpp writes to a new page for the first time, the kernel must intervene to set it up, an event known as a page fault. For large models, saving conversation state involves creating snapshots that can be 150 MB or larger, triggering hundreds of thousands of these faults. Since the thread managing these snapshots also feeds data to the GPU, every pause for a page fault stalls the entire inference pipeline.

Huge pages allow the system to allocate memory in 2 MB blocks instead of 4 KB. By setting the glibc tunable, the application explicitly requests these larger blocks. This drastically reduces the number of times the kernel must stop execution to manage memory metadata. The result is a smoother flow of data to the GPU, as the CPU spends less time handling administrative memory tasks and more time preparing inputs for parallel processing.

Key details

  • The optimization requires glibc version 2.35 or newer, available on Ubuntu 22.04 and Debian 12 or later.
  • Page faults dropped from ~145,000 to ~1,850 per request on H100 hardware during testing.
  • Throughput increased by 21% for tool-calling prompts on H100, but only 6.5% on L4 for the same workload.
  • Ollama users also saw benefits, with performance gains ranging from 2% to 7% depending on the machine.
  • The fix does not significantly benefit frameworks like vLLM or SGLang, which already minimize page faults.
  • Memory usage increased by approximately 0.5 GB on average when running Ollama with this setting enabled.

Why it matters

For developers building products with local LLMs, this finding highlights that GPU power is not the only determinant of inference speed. As GPUs become faster, such as the H100 compared to the L4, the relative cost of CPU-side operations like memory allocation increases. A fixed delay of 0.1 seconds per request represents a much larger percentage of total time on a fast card than on a slower one. This means that optimizing system-level configurations becomes increasingly critical as hardware accelerates.

This approach offers a low-risk, high-reward tuning opportunity for teams using llama.cpp or Ollama. Since it requires no code changes and has minimal impact on stability or answer quality, it can be deployed quickly to improve user experience. The reduction in time-to-first-token is particularly valuable for interactive applications where perceived latency drives user satisfaction.

What you can do

  • Verify your system has glibc 2.35 or newer by running ldd --version in your terminal.
  • Check if transparent huge pages are enabled by inspecting /sys/kernel/mm/transparent_hugepage/enabled.
  • Launch your llama.cpp server with GLIBC_TUNABLES=glibc.malloc.hugetlb=1 prepended to the command.
  • For Ollama, apply the same environment variable when starting the ollama serve process.
  • Monitor effectiveness by checking AnonHugePages in /proc/[pid]/smaps_rollup to ensure huge pages are being used.
  • Compare page fault counts before and after the change using ps -o min_flt= to quantify the reduction.
  • #llama.cpp
  • #linux
  • #performance
  • #ollama
  • #inference

Tools from the Bytechap store

$89

DocBento

Self-hosted document management that reads every scan and answers with page citations.

Live demo

Keep reading

All stories