Ggml 1.0 stabilizes local AI inference with FP8 and multi-backend optimizations
The ggml library reaches version 1.0, introducing FP8 support, YaRN truncation controls, and significant performance improvements across CUDA, Metal, Vulkan, and other backends for local large languag
The ggml open-source library has reached version 1.0.0, marking a major stabilization point for the computational backend that powers many local large language model tools. Released on November 14, 2026, this update removes deprecated APIs, introduces support for 8-bit floating-point precision, and delivers extensive performance optimizations across hardware backends including NVIDIA CUDA, Apple Metal, and Vulkan.
What happened
This release represents a maturation of the ggml codebase, which serves as the foundational tensor library for projects like llama.cpp. The maintainers have pruned the public API by removing legacy operations such as DIAG_MASK_INF and DIAG_MASK_ZERO. This cleanup simplifies the interface for developers and bumps the remote procedure call (RPC) protocol to version 9. The new protocol version enforces stricter compatibility checks, rejecting connections from peers running incompatible software versions, while also improving loading times and handling failed handshakes more gracefully.
Beyond API hygiene, the update introduces critical features for modern model architectures. Developers can now configure YaRN truncation, a technique used to extend the context window of transformers without retraining, through new getter and setter functions. The library also adds native support for OCP FP8 (Open Compute Project 8-bit floating point), a precision format increasingly adopted by hardware vendors to reduce memory bandwidth requirements during inference. These changes are accompanied by broad performance work, including optimized matrix multiplications for small batch sizes and fused operations for Mixture of Experts (MoE) models.
How it works
Ggml operates as a lightweight tensor library designed for efficient inference on consumer hardware. In this release, the core scheduler has been refined to avoid uploading input data that is not read by the current computation graph. It also prefetches only those inputs produced by earlier splits in the execution plan. For MoE models, which activate only a subset of neural network experts for each token, the cache system can now offload some cache miss computations to the CPU and copy results back asynchronously, preventing GPU stalls.
Backend-specific optimizations drive much of the performance gain. On CUDA, the library now supports few-row quantized matrix multiplications for batch sizes between 2 and 24, which are common in interactive chat scenarios. It also implements fused gate and up-matrix multiplications for NVFP4 checkpoints. On Apple’s Metal backend, MoE routing is fused into single command buffers to reduce overhead. Vulkan gains an Intel FlashAttention prefill kernel, while OpenCL introduces decomposed FlashAttention for Adreno GPUs. These changes ensure that the library exploits hardware-specific features like tensor cores or specialized memory layouts without requiring manual tuning from the end user.
Key details
- API Cleanup: Deprecated public APIs and legacy
DIAG_MASKoperations have been removed, simplifying the interface and bumping the RPC protocol to v9. - FP8 Support: Native support for OCP FP8 precision reduces memory bandwidth usage, with specific fused kernels for NVFP4 gate/up matmuls on CUDA.
- YaRN Configuration: New functions
ggml_rope_set_truncateandggml_rope_get_truncateallow developers to control YaRN correction-bound rounding for extended context windows. - Scheduler Efficiency: The core scheduler now skips uploading unused inputs and prefetches only necessary data, reducing memory traffic and latency.
- MoE Optimizations: Fused MoE routing on Metal and a device-side MoE scheduler on SYCL improve throughput for sparse expert models.
- Broad Backend Coverage: Performance improvements span CUDA, Metal, Vulkan, SYCL, OpenCL, WebGPU, Hexagon, and CPU backends, ensuring consistent gains across diverse hardware.
Why it matters
For engineers building local AI applications, ggml 1.0.0 offers a more stable and performant foundation. The removal of deprecated APIs reduces the maintenance burden and minimizes the risk of using outdated or inefficient code paths. The introduction of FP8 support is particularly significant as hardware vendors increasingly prioritize low-precision arithmetic for efficiency. By supporting these formats natively, ggml allows developers to leverage newer GPU capabilities without waiting for higher-level frameworks to catch up.
The performance optimizations address real-world usage patterns. Interactive applications often process small batches of tokens, where traditional large-batch optimizations fail to deliver benefits. The new few-row matrix multiplication kernels and fused MoE operations directly target these scenarios, reducing latency in chat interfaces and coding assistants. Additionally, the improved scheduler logic ensures that memory bandwidth is not wasted on unused data, which is crucial for running large models on devices with limited VRAM.
What you can do
- Update your project dependencies to ggml 1.0.0 to benefit from the stabilized API and performance improvements.
- Test FP8 quantization workflows if your hardware supports it, as this can significantly reduce memory usage during inference.
- Configure YaRN truncation settings using the new API functions if you are working with models that require extended context windows.
- Review your RPC integrations to ensure compatibility with the new v9 protocol, especially if you distribute inference across multiple machines.
- Benchmark small-batch inference tasks to measure the impact of the new few-row matrix multiplication kernels on your specific hardware.
- Monitor MoE model performance, particularly on Metal and SYCL backends, to take advantage of the new fused routing and scheduling features.



