Apple's learned routing cuts RAG latency by replacing centroid search

Researchers at Apple Machine Learning Research have introduced a new method to accelerate maximum inner product search (MIPS), the core engine behind recommendation systems and retrieval-augmented generation (RAG). Published in a paper presented at ICML 2026, the work replaces traditional geometric indexing with small, offline-trained neural networks that learn the specific distribution of user queries. This approach significantly reduces latency while improving recall accuracy compared to standard centroid-based routing.

What happened

Maximum inner product search is computationally expensive when databases contain millions of high-dimensional vectors. Traditional approximate methods speed up this process by clustering keys and using centroids to route queries, but these geometric assumptions often fail because user queries differ structurally from the curated documents they seek. Apple’s team observed that generic, query-agnostic methods ignore the stable, well-characterized distribution of queries in production environments, leaving performance gains untapped.

To address this, the researchers trained small neural networks offline on a fixed database of keys. These networks learn to map queries to their best matches more efficiently than exhaustive comparison or simple centroid routing. The team evaluated two architectures, SupportNet and KeyNet, using the Natural Questions benchmark, which pairs general-knowledge questions with roughly 2.6 million unique passages. The results demonstrated that learned routers could achieve higher recall at lower latency by leveraging the specific geometry of the query distribution rather than relying on broad geometric approximations.

How it works

The core insight relies on viewing MIPS through the lens of convex analysis. The function that maps a query to its largest inner product with any key in a set is known as a support function. This function is convex, meaning it can be approximated by input convex neural networks (ICNNs). Furthermore, the gradient of this support function at any given query point returns the best-matching key vector. By training a neural network to mimic this support function or its gradient, the system can predict optimal matches with far less computational cost than searching the entire database.

The researchers developed two specific models to exploit this mathematical structure. SupportNet learns the support function directly for each data cluster, allowing it to answer how well the best key inside a cluster matches a query. This enables more accurate routing than centroid distance alone. KeyNet, conversely, learns to output the best-matching key vector directly. It acts as a query rewriter, transforming the original query into a vector that is closer to the target in embedding space before passing it to a standard search index.

Key details

• The research was presented at ICML 2026 by Apple Machine Learning Research.
• SupportNet models achieved 95% routing recall 1.9 times faster than centroid routing (1.74 ms vs 3.36 ms).
• KeyNet reduced the time to reach 99% recall by up to 8 times compared to raw query search (0.23 ms vs 1.8 ms).
• The models were trained on the Natural Questions dataset containing approximately 2.6 million unique passages.
• SupportNets were tested in sizes ranging from 1% to 20% of the database size, with depths of 16 layers.
• KeyNet-transformed queries reached 99% recall while scanning only 0.009% of the database, compared to 0.10% for raw queries.

Why it matters

For engineers building RAG systems, the gap between short user questions and long document embeddings creates a persistent bottleneck. Standard inverted file indexes rely on centroids that do not account for the spread of keys within a cluster, leading to missed neighbors and unnecessary compute waste. By learning the specific query distribution, developers can redirect compute resources more efficiently, ensuring that the most relevant documents are retrieved without exhaustive searches.

This method also offers a practical path to integrating with existing infrastructure. KeyNet can function as a pre-processing step for standard approximate search indices like FAISS, requiring no changes to the underlying storage system. This means teams can achieve significant latency reductions and accuracy improvements by adding a lightweight neural network layer in front of their current retrieval pipeline, rather than rebuilding their entire vector search architecture.

What you can do

• Evaluate whether your production query distribution is stable and well-characterized, as this method relies on offline training on representative data.
• Consider implementing SupportNet to replace centroid-based routing in your inverted file index if routing errors are impacting recall.
• Use KeyNet as a query rewriting layer before your existing vector search index to reduce the number of keys scanned during retrieval.
• Train small neural networks offline using regression on sampled queries and their ground-truth best matches to amortize the computational cost.
• Benchmark the trade-off between model size and latency, noting that even small models (1% of database size) offer significant speedups over baseline methods.
• Test the consistency loss derived from Euler’s theorem when training KeyNet models to keep predictions geometrically grounded.