Building with AI

Token pooling-aware training cuts vector storage while keeping search quality

A new training method teaches multi-vector encoders to preserve retrieval signals after compressing document vectors, reducing storage by 75% with minimal quality loss.

Illustration of vector pooling compressing data points
Illustration generated for this article

Hugging Face published a technical guide on November 11, 2026, detailing a method called token pooling-aware training for multi-vector embedding models. The approach addresses the high storage costs of models like ColBERT by teaching encoders to maintain retrieval accuracy even after aggressively merging similar token vectors.

What happened

Multi-vector retrieval models assign a separate vector to each token in a document, allowing different parts of a query to match specific sections of text. While this yields high expressiveness, it creates massive storage requirements. A corpus of one million documents can easily generate hundreds of millions of vectors, consuming gigabytes of space even with quantization.

To reduce this footprint, developers often use token pooling, which merges similar token vectors into single representations. However, standard pooling is applied after training, meaning the model never learned to handle the information loss caused by merging. This often leads to degraded search quality, as subtle distinctions between tokens are erased.

The new guide demonstrates how to integrate pooling directly into the training loop. By optimizing the model to perform well on both original and pooled representations simultaneously, the encoder learns to cluster related concepts in a way that survives compression. The result is a model that retains nearly all its retrieval power while storing significantly fewer vectors per document.

How it works

The core mechanism relies on Hierarchical Token Pooling, which uses Ward hierarchical clustering to group similar token embeddings. Starting with individual tokens, the algorithm repeatedly merges the two groups that cause the smallest increase in internal variance. This creates a hierarchy of clusters, allowing developers to choose a compression level, such as keeping only one-quarter of the original vectors.

During training, the system applies a combined loss function. Half of the loss comes from retrieving with full, unpooled embeddings, ensuring the model learns rich representations. The other half comes from retrieving with pooled embeddings, forcing the model to arrange tokens so that critical information remains accessible after merging. A straight-through estimator allows gradients to flow back through the discrete clustering decisions, updating the encoder weights accordingly.

This process is implemented using Sentence Transformers and a custom loss wrapper. The training script calculates embeddings once per batch but applies multiple pooling strategies to those same embeddings, adding minimal computational overhead. Developers can also accelerate the Ward clustering and MaxSim scoring steps using CUDA kernels, which offer significant speedups over CPU-based implementations.

Key details

  • Storage reduction: Using a pool factor of 4 reduces document vectors by approximately 75%, cutting storage from 12.8 GB to 3.2 GB for a million-document corpus at 4-bit precision.
  • Quality retention: The pooling-aware model retains 97.8% of its unpooled NanoBEIR nDCG@10 score at a pool factor of 4, compared to 94.6% for a standard model.
  • Training method: The loss function averages retrieval scores from unpooled and pooled representations, teaching the encoder to be robust to compression.
  • Flexibility: Pooling can be controlled by a fixed factor, a maximum token count, or an adaptive distance threshold calibrated on a sample corpus.
  • Performance: CUDA kernels for Ward clustering can be 30 to 150 times faster than SciPy implementations, depending on settings.
  • Baseline improvement: Even unpooled retrieval quality improves slightly with this training method, rising from 51.84 to 52.39 mean nDCG@10.

Why it matters

For engineers building retrieval-augmented generation (RAG) systems, storage cost and latency are constant constraints. Multi-vector models offer superior accuracy for complex queries but are often prohibitive at scale due to their memory footprint. This technique allows teams to deploy high-performance multi-vector retrievers without the associated infrastructure bill, making advanced search feasible for larger datasets.

The method also simplifies the trade-off between speed and accuracy. By training the model to expect compression, developers can adjust the pooling factor at inference time without retraining. This means a single model can serve different use cases, from high-precision archival search to low-latency real-time suggestions, simply by changing the pooling configuration.

Furthermore, the approach is compatible with other compression techniques like quantization and Matryoshka embeddings. Teams can layer these methods to achieve extreme compression ratios while maintaining acceptable retrieval quality, providing a flexible toolkit for optimizing search infrastructure.

What you can do

  • Install the required dependencies using pip install -U "sentence-transformers[train,kernels]" to access the training scripts and CUDA accelerations.
  • Run the provided example script with uv run to train a baseline model and a pooling-aware model on the MS MARCO dataset for comparison.
  • Calibrate an adaptive pooling threshold on your own document corpus using HierarchicalTokenPooling.calibrate_distance_threshold to find the optimal balance for your data.
  • Evaluate your trained models on NanoBEIR or a domain-specific benchmark to measure the impact of different pool factors on nDCG@10.
  • Enable CUDA kernels during inference by setting use_kernels=True when loading the MultiVectorEncoder to speed up clustering and scoring.
  • Experiment with combining token pooling with 4-bit quantization to further reduce storage requirements for large-scale deployments.
  • #machine learning
  • #vector search
  • #rag
  • #embeddings
  • #optimization

Tools from the Bytechap store

Keep reading

All stories