Skip to content
Blog

How to Benchmark Vector Databases Honestly

A checklist for benchmarking vector search fairly: ground truth, real queries, comparable settings, and the traps that make vendor numbers meaningless.

Published on • November 22, 2026

AI Assistant

Introduction

“10× faster than the competition” is the most common claim in the vector database industry — and the least meaningful without methodology. Honest benchmarking means controlling ground truth, query sets, hardware, and settings so the numbers actually compare. Here is how to do it.

Step 1: Define the Workload

Before touching a database, pin down:

  • Corpus: size, dimensionality, metric (cosine/IP/L2), data type (float32/int8/PQ)
  • Queries: real query distribution, k (top-10 vs top-100), filter selectivity
  • Load: target QPS, concurrency, latency SLO (p95/p99)
  • Environment: CPU vs GPU, RAM vs disk, cloud instance type

A benchmark on 1M random GloVe vectors says little about your 50M-vector filtered RAG workload.

Step 2: Build Exact Ground Truth

Compute the true top-k for every query with brute force (or a verified exact engine). Every recall number is returned ∩ truth / k. Without ground truth you can measure speed, but you cannot measure quality — and speed without quality is meaningless.

Step 3: Compare Like with Like

Vendor benchmarks drift apart on settings that silently change results:

FairUnfair
Same recall@k when comparing latencyComparing latency at different recall
Same hardware, thread count, RAMOne system on NVMe, another in RAM
Same build-time budgetOne index tuned for hours, one left default
Includes index build + memory footprintMemory omitted entirely
Real inserts/updates workloadStatic index only, if yours is dynamic

The canonical fair view is the recall-vs-latency curve: sweep each system’s quality knob (efSearch, nprobe, …) and plot recall@k against p95 latency. A system that sits on the curve’s frontier wins — not the one with a single cherry-picked point.

Step 4: Measure the Full Picture

Track at least:

  • recall@k vs exact ground truth
  • p50 / p95 / p99 latency — not averages
  • QPS at the latency target (and latency at your production QPS)
  • Index build time and memory/disk footprint
  • Insert/update throughput if data is mutable
  • Cost — performance per dollar on the hardware you’ll actually run

Step 5: Avoid the Classic Traps

  1. Random vectors are toy data — real embeddings cluster and skew; clustering changes IVF/HNSW behaviour.
  2. Warm-cache only numbers — report cold too if your working set exceeds RAM.
  3. Ignoring filters — selective metadata filters change the game completely (pre-filter vs post-filter).
  4. Single-k reporting — recall@1 and recall@100 can rank systems differently.
  5. No repeats — one run’s variance can fake a 10% win; repeat and report spread.
  6. Benchmarking the demo config — defaults are tuned for the demo, not your data.

The ANN-Benchmarks Discipline

The ANN-Benchmarks project (ann-benchmarks.com) standardises exactly this: fixed datasets, shared ground truth, a recall-vs-QPS plot per distance measure, plus index size and build time. Use it to shortlist algorithms; then re-run the methodology against your chosen databases on your own data.

Conclusion

An honest benchmark is a workload definition, an exact ground truth, matched settings, and full curves instead of point claims. Do that, and “10× faster” becomes something you can verify yourself — or debunk in an afternoon.

References