Google's TPU Sync reduces inference latency by moving KV-cache data

Google has introduced TPU Sync, a new open-source software framework designed to accelerate AI inference on its eighth-generation Tensor Processing Units. Announced on November 9, 2026, the tool addresses memory bottlenecks in large language model serving by optimizing how key-value cache data moves between different memory tiers. The company claims this approach can reduce time to first token by more than 90 percent in specific high-parameter scenarios.

What happened

The release accompanies Google’s broader hardware updates, specifically targeting developers who run large models with million-token context windows or reinforcement-learning workloads. As context lengths grow, the cost and latency of managing the key-value (KV) cache become significant bottlenecks. TPU Sync aims to solve this by efficiently moving data between accelerator memory, host DRAM, storage, and the network without unnecessary copying.

In internal benchmarks, Google demonstrated substantial performance gains. For the 235-billion-parameter Qwen3 model, moving KV caches dropped from several seconds to approximately 230 milliseconds. This reduction lowered the average time to first token by 60 percent under light load. When testing a larger 480-billion-parameter model, the latency reduction reached 90 percent. Similar improvements were observed in prefix caching tests with 64,000-token contexts, where throughput increased by over 50 percent and P99 inter-token latency fell by 30 percent.

However, these results come with caveats. Performance varies based on the specific workload, the amount of cached data reuse, and available network bandwidth. The software is not yet considered production-ready for all use cases. Google warns that the APIs are still evolving and advises developers to pin their deployments to specific verified commits or tagged releases rather than tracking the main branch.

How it works

TPU Sync is written in C++ and operates by managing data transfers across the memory hierarchy of TPU hosts. It utilizes zero-copy transfers and native PjRtBuffer hardware descriptors to eliminate redundant data duplication. This method allows the system to approach the theoretical limits of the hardware interface. In tests on Ironwood TPU hosts, combined host-to-device and device-to-host throughput reached 223 GB/s, nearing the 256 GB/s limit of PCIe Gen5.

The framework integrates with existing inference ecosystems, including TorchTPU, vLLM, and SGLang. This compatibility means developers can run disaggregated workloads on TPUs without rebuilding their entire serving stack. Disaggregation separates the prefill and decode phases of inference, allowing each to be optimized independently. Previously, external TPU serving lacked the infrastructure to handle this separation effectively, but TPU Sync provides the necessary mechanisms to manage state and data movement between these phases.

Key details

• TPU Sync is an open-source framework written in C++ for managing KV-cache data movement.
• Benchmarks show up to 90% reduction in time to first token for 480-billion-parameter models.
• Throughput on Ironwood TPU hosts reached 223 GB/s, close to the PCIe Gen5 theoretical limit.
• The software supports Cloud TPU v7x and v6e with specific JAX and PyTorch version requirements.
• Integration is available for popular inference engines like vLLM and SGLang via TorchTPU.
• Google advises using verified commits or tagged releases as the APIs are still changing.

Why it matters

For engineering teams building products with large language models, inference latency directly impacts user experience and operational costs. Time to first token is a critical metric for interactive applications, and reducing it by such margins can make previously impractical context windows viable. By addressing the software layer that manages memory movement, Google is attempting to close the gap with Nvidia’s CUDA ecosystem, which has long dominated AI inference due to its mature tooling and performance optimizations.

This development also highlights the growing importance of disaggregated serving architectures. As models scale, treating prefill and decode as separate problems allows for better resource utilization. TPU Sync provides the infrastructure to make this possible on Google hardware, potentially offering a cost-effective alternative to Nvidia’s GB200 and GB300 systems for certain workloads. However, direct comparisons with fully tuned Nvidia setups remain limited, and developers must weigh the benefits against the current immaturity of the software.

What you can do

• Evaluate if your current workloads involve large context windows or high-parameter models that would benefit from KV-cache optimization.
• Test TPU Sync in a non-production environment using Cloud TPU v7x or v6e instances to measure actual latency improvements.
• Pin your dependencies to specific tagged releases or verified commits to avoid breaking changes from evolving APIs.
• Review your existing inference stack to see if it can integrate with TorchTPU, vLLM, or SGLang for easier adoption.
• Monitor network bandwidth and cached data reuse rates, as these factors significantly influence the performance gains you will see.
• Keep an eye on future updates from Google regarding API stability and broader hardware support before planning production migration.