Hybrid Search for Agent Context: Dense, Sparse, and Keyword

Your agent gets the query: “Did we ship order #ORD-2025-09847 to the customer who bought 500 units of SKU-X in Q3?” Dense retrieval finds similar concepts but blurs the exact order number. BM25 nails the order number but can’t connect “payment processor” to “Stripe API key.” You need both.

Why Neither Dense nor Sparse Alone Suffices

Dense (semantic) retrieval excels at paraphrase matching and conceptual similarity. “Refund status” matches “money returned.” But it blurs exact identifiers — product codes, error codes, API names get flattened into generic embedding vectors.

Sparse (BM25) retrieval excels at exact-match queries. But it has no notion of synonymy — it can’t connect “payment processor authentication” to “Stripe API key” when vocabulary doesn’t overlap.

As one production guide puts it: “Picking either method alone means leaving recall on the table, and no downstream re-ranking can rescue a chunk that was never retrieved.”

Benchmark Evidence

Setup	NDCG@10	vs. Best Single
BM25 only	0.515	—
Dense only	0.466	—
Hybrid RRF fusion	0.551	+7%
Hybrid + Cohere Rerank	0.683	+33%

Hybrid RRF + Cross-Encoder reranking achieves 0.81 NDCG@5 — a 16-point improvement over unranked hybrid.

The Three-Stage Pipeline

Query
  ├─→ Sparse Retrieval (BM25) ──┐
  └─→ Dense Retrieval (ANN) ────┤
                                 ├─→ Fusion (RRF) ──→ Reranking ──→ Top-k

Stage 1: Parallel Retrieval

Both indexes queried simultaneously, returning top-50 to top-500 candidates. Running in parallel adds minimal latency (10-50ms over single retrieval).

Stage 2: Score Fusion with RRF

BM25 scores and cosine similarity live on incompatible scales. Reciprocal Rank Fusion solves this by operating on ranks:

RRF_score(d) = Σ 1/(k + rank_i(d))

The constant k=60 is standard. RRF requires no tuning and is the default in Elasticsearch, Qdrant, and Weaviate.

Alternative fusion methods:

• Relative Score Fusion — MinMax normalizes scores, weighted sum (~6% better recall)
• Convex Combination — Weighted sum with alpha parameter
• Distribution-Based — Scales by mean/stddev of each result set

Stage 3: Cross-Encoder Reranking

A second-stage precision pass. Cross-encoders process (query, candidate) pairs jointly through transformer layers. 50-100x slower but far more accurate — applied only to the 20-50 fused candidates, not the full corpus.

Key reranking models: cross-encoder/ms-marco-MiniLM-L-6-v2, BAAI/bge-reranker-large, Cohere Rerank API.

“Don’t increase Top-K to give the LLM more information. 3-5 highly-ranked chunks produce better answers than 15 poorly-sorted ones.”

Why Agents Specifically Need Hybrid

Agents query over heterogeneous data — CRM records, support tickets, code repos, compliance documents — mixing semantic intent with exact identifiers. The retrieval needs to handle both.

Agentic Hybrid Retrieval Architecture

1. Planner Agent decomposes queries, generates sub-queries and expansions
2. Retriever Agent executes hybrid search (BM25 + dense + RRF)
3. Evaluator Agent scores result sufficiency, triggers refinement
4. Reranker Agent performs final listwise reranking

Implementation

Python: In-Memory Hybrid Search

from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder
import numpy as np

# Ingestion
bm25 = BM25Okapi(tokenized_corpus)
dense_index = VectorStore(corpus_embeddings)

# Query
def hybrid_search(query, top_k=20):
    dense_results = dense_index.search(embed(query), top_k=top_k)
    sparse_results = bm25.search(tokenize(query), top_k=top_k)
    
    # Reciprocal Rank Fusion
    rrf_scores = {}
    k = 60
    for rank, doc in enumerate(dense_results):
        rrf_scores[doc.id] = rrf_scores.get(doc.id, 0) + 1.0 / (k + rank + 1)
    for rank, doc in enumerate(sparse_results):
        rrf_scores[doc.id] = rrf_scores.get(doc.id, 0) + 1.0 / (k + rank + 1)
    
    fused = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return [doc_id for doc_id, _ in fused]

# Cross-encoder reranking
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query, candidates, top_k=5):
    pairs = [(query, doc["text"]) for doc in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, _ in ranked[:top_k]]

LlamaIndex Integration

from llama_index.core.retrievers import QueryFusionRetriever

retriever = QueryFusionRetriever(
    retrievers=[vector_retriever, bm25_retriever],
    num_queries=4,  # Generate expansion queries
    use_async=True,
    fusion_mode="reciprocal_rerank",
)

Key Design Decisions

Decision	Recommendation	Rationale
Fusion method	Start with RRF (k=60)	No tuning, robust across corpora
Candidate count	50-100 from fusion, 5-10 to LLM	Beyond 50, reranking latency increases
Reranker	Cross-encoder (MiniLM-L6)	15-20 point NDCG improvement
Chunking	Recursive, 256-512 tokens, with overlap	Works for both sparse and dense

When Hybrid Is NOT Necessary

• Corpus is uniformly conceptual (dense alone suffices)
• Corpus is uniformly keyword-driven (BM25 alone suffices)
• Sub-50ms p95 retrieval is required and the second retriever blows the budget

The Takeaway

Hybrid search is the default for any agent that queries heterogeneous data. Start with RRF fusion (no tuning needed), add a cross-encoder reranker for the final top-k, and you’ll see 15-30% retrieval quality improvement over single-method approaches.

💡 Weaviate’s hybrid query with relativeScoreFusion is the easiest production entry point — native support with minimal configuration.