llama.cpp v1.0 delivers major speed gains for parallel and speculative decoding
The llama.cpp v1.0 release introduces breaking API changes and significant performance improvements for multi-request and speculative decoding workloads on NVIDIA RTX 30-series GPUs.
On November 13, 2026, the llama.cpp project reached version 1.0, marking a shift to strict semantic versioning for its C/C++ inference engine. Benchmarks conducted on November 14 using an NVIDIA RTX 3090 reveal that while single-user performance remains stable, parallel processing and speculative decoding see substantial speed increases of up to 55 percent.
What happened
The release of llama.cpp v1.0.0 (build 12359) introduces several breaking changes to the public C API and command-line interface. The most immediate operational change for users is the removal of the --defrag-thold flag, which must be stripped from startup scripts before updating. Additionally, security defaults have tightened: the llama-server now refuses to enable built-in tools or Model Context Protocol (MCP) servers on non-loopback network addresses unless an API key is provided via --api-key. Requests referencing private or local media URLs are also blocked by default unless explicitly allowed.
Performance testing compared this new build against a pre-v1 version (build 12199) from five days earlier. The tests ran on Fedora 44 with Docker 29.9.2, using an AMD Ryzen 9 5900X CPU and an NVIDIA RTX 3090 GPU with 24 GB of VRAM. Models tested included Qwen3.5 9B and 27B in Q4_K_M quantization. The results indicate that for single-user chat sessions without advanced features, performance is virtually identical to previous builds. However, scenarios involving multiple concurrent requests or speculative decoding show marked improvements.
How it works
The primary performance gain in v1.0 comes from a new CUDA kernel designed for NVIDIA RTX 30-series (Ampere) and newer GPUs. This kernel optimizes operations where the model processes between 2 and 24 tokens simultaneously. In standard single-request mode, the model processes one token per step, utilizing an older kernel that remains unchanged. However, when handling parallel requests or speculative decoding, the workload shifts to multi-token steps.
Speculative decoding uses a smaller draft model to guess subsequent tokens, which the larger main model then verifies in a single step. This verification process involves checking multiple tokens at once, directly leveraging the new kernel’s ability to read weights once and multiply them across all tokens in the step. Similarly, parallel requests use continuous batching to process tokens from different users together. Because these workflows align with the new kernel’s design, they see significant throughput increases, while single-token sequential processing does not.
Key details
- Parallel throughput: Handling four simultaneous requests resulted in a 14% speed increase for the 9B model and a 22% increase for the 27B model compared to the pre-v1 build.
- Speculative decoding: Using Multi-Token Prediction (MTP) with the 27B model yielded a 52% to 55% increase in tokens per second, rising from roughly 45 tok/s to 68.5 tok/s.
- Single-user performance: For a single request without speculative decoding, speed differences were negligible, staying within a 1.5% margin of error in both directions.
- Breaking changes: The
--defrag-tholdflag is removed. Networked servers require an API key to use tools, and private URL access requires the--media-allow-private-urlsflag. - New features: The release includes an FP8 option for context cache, a system RAM cache for mixture-of-experts models, and a text-to-speech endpoint at
/v1/audio/speech. - Versioning policy: Future major version bumps will only occur for breaking API or ABI changes, providing more stability expectations for developers building on top of llama.cpp.
Why it matters
For developers and technical leads managing local LLM infrastructure, this release signals a maturation of the toolchain. The move to semantic versioning means that integrating llama.cpp into production applications becomes less risky, as breaking changes are now clearly signaled by major version numbers. However, the immediate impact is operational: teams must update their deployment scripts to remove deprecated flags and configure API keys for any network-exposed instances. Failure to do so will result in startup errors or disabled functionality.
The performance implications are specific to workload types. If you run a personal chatbot for single-user use, the upgrade offers no urgent performance benefit. However, for services supporting multiple concurrent users, agents, or coding assistants that hit the server simultaneously, the 14-22% throughput gain is significant. Even more impactful is the boost to speculative decoding, which makes running larger, more capable models like Qwen3.5 27B much more responsive on consumer hardware. This makes high-quality local inference more viable for interactive applications where latency matters.
What you can do
- Update startup scripts: Remove the
--defrag-tholdor-dtflag from yourllama-servercommands before upgrading to v1.0. - Secure networked instances: If your server listens on a non-localhost address, add an
--api-keyto prevent tools and MCP servers from being disabled by default. - Enable private media access: If your workflow involves images or media hosted on local network addresses, add the
--media-allow-private-urlsflag to maintain compatibility. - Test parallel loads: If you serve multiple clients, benchmark your specific hardware with the new build to quantify the 14-22% throughput gain for your model size.
- Experiment with speculative decoding: Try enabling MTP or speculative decoding with larger models to leverage the 50%+ speedup on RTX 30-series and newer GPUs.
- Check dependency compatibility: If you build software on top of llama.cpp’s C API, review the release notes for breaking changes before adopting the v1.0 headers.



