Restoring K80 performance for quantized LLMs with a custom llama.cpp patch

A new open-source patch addresses a critical performance bottleneck for users running large language models on older NVIDIA Kepler architecture GPUs. Released in November 2026, this fix targets the llama.cpp library, specifically resolving slow inference speeds for K-quantized models on hardware like the Tesla K80. The solution restores near-native performance by correcting how non-contiguous memory blocks are handled during dequantization.

What happened

Users of legacy GPU hardware, particularly those with Compute Capability below 5.0 such as the Tesla K80, experienced severe slowdowns when running K-quant models (q2_K through q6_K). The issue stemmed from how the llama.cpp library routed matrix multiplication tasks. When dealing with non-contiguous K-quants, the system lacked a specific non-contiguous dequantize converter, often referred to as an "nc-converter." Without this component, the software fell back to a slower execution path known as MMQ or offloaded work entirely to the CPU.

The MMQ path on Kepler GPUs is approximately four times slower than the standard cuBLAS implementation. This performance gap made local inference impractical for many enthusiasts and developers relying on affordable, older hardware. The bottleneck was not due to raw compute limits but rather a missing software bridge that prevented efficient data handling between memory and the GPU cores.

To address this, a developer created a specialized patch available in a public repository. Although an upstream pull request (#30338) was closed by maintainers due to procedural concerns regarding submission timing and evidence updates, the code remains accessible for those who need it. The patch was developed through careful measurement and iteration, involving both human analysis and automated agents, ensuring technical validity despite its exclusion from the main branch.

How it works

The fix introduces three primary components to correct the data flow. First, a "cont-cpy" mechanism adjusts how memory blocks are copied. It uses whole-block walking based on ggml_type_size rather than individual element sizes, ensuring that contiguous operations for formats like q4_K function correctly. This prevents fragmentation issues that previously forced inefficient processing paths.

Second, the patch adds a stride-enabled dequantize kernel for fp32 output, covering all K-quant variants from q2 to q6. This kernel is registered within the CUDA backend, allowing the GPU to directly handle non-contiguous data without falling back to slower methods. A third component extends this capability to fp16 and bf16 outputs, ensuring broad compatibility with different model precision requirements.

Finally, the routing logic in ggml_cuda_should_use_mmq is updated. It now checks for the presence of an nc-converter before defaulting to the slow MMQ path. If a converter exists, even for non-contiguous source data, the system routes the task to cuBLAS. This ensures that contiguous K-quants always use the fast path, while non-contiguous ones benefit from the new kernels.

Key details

• Hardware target: NVIDIA Kepler GPUs (Compute Capability < 5.0), such as the Tesla K80.
• Performance gain: Prefill speed increases from 42.8 tokens per second to approximately 140 tokens per second, a 3.3x improvement.
• Accuracy: Normalized Mean Squared Error (NMSE) is exactly 0 for all five K-quants in both fp16 and bf16 modes.
• Regression testing: No performance regression observed on RTX 3090 (CUDA0) or K80 (CUDA2) test suites.
• CPU comparison: The GPU implementation remains roughly twice as fast as CPU execution across all batch sizes.
• Availability: The patch is hosted in a separate repository after the upstream PR was closed for procedural reasons.

Why it matters

For developers and hobbyists building local AI infrastructure, hardware longevity is a key economic factor. Many still rely on used enterprise cards like the K80 due to their high memory bandwidth and low cost. This patch effectively extends the useful life of these devices for modern LLM workloads. Without it, the inefficiency of the MMQ path renders these GPUs nearly obsolete for quantized model inference, forcing users to upgrade or accept unusable latency.

The situation also highlights the complexity of maintaining high-performance libraries across diverse hardware generations. As llama.cpp evolves to support newer quantization formats and acceleration methods, edge cases for older architectures can emerge. This fix demonstrates that significant performance gains often come from optimizing memory access patterns and data conversion routines rather than just raw compute optimization. It serves as a reminder that software routing logic is as critical as kernel efficiency.

What you can do

• Identify if your GPU has Compute Capability below 5.0 to determine if you are affected by this bottleneck.
• Test the provided patches (patch-llama-cuda-kepler-cont-cpy.patch and patch-llama-cuda-kepler-nc-dequant-kquants.patch) on your local llama.cpp build.
• Measure prefill and generation speeds before and after applying the fix to verify the 3.3x improvement claim.
• Ensure you are using K-quant models (q2_K to q6_K) to see the specific benefits of the new dequantize kernels.
• Monitor NMSE values during testing to confirm that the zero-error correctness claim holds for your specific workload.
• Keep track of upstream llama.cpp developments, as future versions may integrate similar fixes or change routing logic further.