Vector database benchmarks reveal wide gaps in build time and concurrency

Percona released a rigorous benchmark comparing vector search performance across six database engines, highlighting significant disparities in indexing speed, memory usage, and concurrency handling. The study evaluated MariaDB, AliSQL, PostgreSQL with pgvector, Valkey, and Percona Search for MongoDB using a corpus of 990,000 OpenAI embeddings. Results show that while some engines excel at rapid index construction, others offer superior storage efficiency or high-concurrency throughput, providing critical data for engineers designing retrieval-augmented generation (RAG) systems.

What happened

The benchmark loaded 990,000 vectors of 1,536 dimensions into each engine, built an HNSW index, and measured five key metrics: time to create a queryable index, storage and memory footprint, recall versus throughput, performance under concurrent clients, and behavior during data churn. All tests ran on identical hardware, an Intel Xeon Gold 6230 server with 80 logical CPUs and 187.5 GB RAM, ensuring a fair comparison. The orchestrator applied consistent resource limits, such as a 42.3 GB InnoDB buffer pool and specific memory allocations for graph caches, to prevent configuration bias.

The results revealed unexpected variations in basic operations. Time to build a queryable index ranged from just 1.9 minutes for Percona Search to 6.7 hours for AliSQL. Storage efficiency also varied widely, with MariaDB producing the smallest index at 3.9 GiB, while Percona Search required 15.1 GiB. Notably, the engine with the smallest index took the longest to build it, illustrating a common trade-off between compression and construction speed. These findings suggest that choosing a vector database requires balancing immediate operational needs, such as re-indexing frequency, against long-term storage costs.

Concurrency testing exposed further distinctions. Under a load of 32 concurrent clients, PostgreSQL with pgvector scaled its throughput by 21.7 times compared to single-client performance. In contrast, the three InnoDB-based engines—MariaDB 11.8, MariaDB 12.3, and AliSQL—showed almost no scaling, maintaining roughly 1x throughput. However, the benchmark team withdrew specific concurrency efficiency numbers for these three engines after discovering a limitation in the test harness. The MariaDB Connector/C held the Python interpreter lock during socket waits, meaning some clients were waiting on the lock rather than the database, skewing the "in service" metrics.

How it works

The benchmark used the dbpedia-openai-1000k-angular dataset, consisting of 990,000 float32 vectors. Each engine built an HNSW index with M=16, a standard parameter controlling the connectivity of the graph structure. Query performance was measured using different ef_search values, which determine the number of candidates explored during search. Recall was calculated using the distance-threshold definition from ann-benchmarks, ensuring comparability with existing literature. For filtered queries, ground truth was recomputed via brute force over only the rows satisfying the predicate, preventing engines from being penalized for correctly excluding non-matching items.

Different engines employ distinct strategies for filtered search. PostgreSQL with pgvector uses a post-filter approach where it collects candidates from the graph and then drops those failing the predicate. If too many are dropped, it stops early, potentially returning fewer results than requested. Percona Search uses a pre-filter strategy, which can maintain higher throughput as filters tighten but may sacrifice recall. MariaDB and AliSQL also post-filter but continue searching, which preserves recall but can significantly increase latency as the filter selectivity decreases. These architectural choices directly impact how well each engine handles complex, real-world queries involving metadata filters.

Key details

• Index build time: Percona Search was fastest at 1.9 minutes, while AliSQL was slowest at 6.7 hours.
• Storage efficiency: MariaDB 11.8 and 12.3 produced the smallest indexes at 3.9 GiB, compared to 15.1 GiB for Percona Search.
• Single-client throughput: At 0.99 recall, MariaDB 12.3 led with 277 QPS, followed by Valkey at 264 QPS and PostgreSQL at 210 QPS.
• Concurrency scaling: PostgreSQL throughput increased 21.7x with 32 clients, whereas InnoDB-based engines showed minimal scaling due to server-side queuing.
• Filtered search: PostgreSQL’s default post-filter method caused recall to drop to 0.1034 at 1% selectivity because it stopped early after discarding non-matching candidates.
• Data churn: MHNSW and VIDX update the graph on every insert, making their load time equivalent to index build time, unlike pgvector which separates copying and building.

Why it matters

For software engineers building AI applications, these benchmarks highlight that no single engine dominates all categories. If your application requires frequent re-indexing or model updates, the difference between a 40-minute build time for PostgreSQL and a 6.7-hour window for AliSQL is operationally significant. Conversely, if storage cost is the primary constraint, MariaDB’s compact 3.9 GiB index offers substantial savings over other options, despite its slower build process. Understanding these trade-offs helps teams select infrastructure that aligns with their specific workload patterns, whether they prioritize write speed, read throughput, or storage density.

The concurrency results are particularly relevant for high-traffic services. The ability of PostgreSQL and Valkey to scale throughput linearly with client count suggests they are better suited for environments with many simultaneous users. The flat performance of InnoDB-based engines under concurrent load indicates potential bottlenecks in their current vector implementations or connector libraries. Engineers must account for these scaling characteristics when designing system architecture, potentially requiring additional load balancing or connection pooling strategies for engines that do not scale well natively.

What you can do

• Evaluate your re-indexing frequency; if you update embeddings often, prioritize engines with faster build times like PostgreSQL or Valkey.
• Test filtered search performance with your specific metadata distribution, as default configurations may lead to poor recall or high latency.
• Consider storage costs versus build time; if disk space is expensive, MariaDB’s smaller indexes may justify longer build windows.
• Benchmark concurrency for your expected user load, keeping in mind that some engines may require more instances to handle parallel queries effectively.
• Review connector libraries for language-specific clients, as harness limitations in this study suggest driver-level bottlenecks can mask database performance.
• Monitor memory usage during index builds, as peak RSS varied from 14.3 GB to 48.8 GB, impacting infrastructure provisioning requirements.