Re-Ranking Retrieved Context Before Injection: Precision Over Recall in RAG

Retrieval gets you candidate documents. Re-ranking gets you the right ones. In RAG pipelines, the initial retrieval casts a wide net—re-ranking narrows it to the documents that actually matter. This guide covers practical re-ranking techniques that improve answer quality without changing your vector store.

The Problem with Initial Retrieval

Vector similarity search returns documents based on embedding distance. This has limitations:

1. Embedding models are bi-encoders: They encode query and document independently, missing fine-grained interactions
2. Top-K is arbitrary: The 5th result might be more relevant than the 1st
3. No diversity: Results often contain near-duplicates that waste context window space
4. Semantic gap: Embedding similarity ≠ answer relevance

Re-ranking addresses all of these.

Re-Ranking Architecture

User Query
    ↓
[Initial Retrieval] → Top-K candidates (K=20-50)
    ↓
[Cross-Encoder Re-Ranking] → Precision scoring
    ↓
[Diversity Filtering] → Remove duplicates
    ↓
[Top-N Selection] → Final context (N=3-5)
    ↓
[LLM Generation]

Technique 1: Cross-Encoder Re-Ranking

Cross-encoders process query and document together, achieving higher accuracy than bi-encoders:

from sentence_transformers import CrossEncoder

# Load a pre-trained cross-encoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

def rerank_cross_encoder(query: str, documents: list[str], top_n: int = 5):
    """Re-rank documents using cross-encoder."""
    # Create query-document pairs
    pairs = [(query, doc) for doc in documents]
    
    # Score all pairs
    scores = reranker.predict(pairs)
    
    # Sort by score
    ranked = sorted(
        zip(documents, scores),
        key=lambda x: x[1],
        reverse=True
    )
    
    return ranked[:top_n]

# Example
documents = [
    "Python is a programming language",
    "How to deploy Python apps to AWS",
    "Python data structures tutorial",
    "AWS Lambda pricing model",
    "Deploying machine learning models",
]

reranked = rerank_cross_encoder(
    "how to deploy Python to cloud",
    documents,
    top_n=3
)
# [
#   ("How to deploy Python apps to AWS", 8.92),
#   ("Deploying machine learning models", 6.45),
#   ("AWS Lambda pricing model", 4.21),
# ]

Why Cross-Encoders Work Better

Approach	Query Encoding	Document Encoding	Interaction
Bi-Encoder	Separate	Separate	Dot product
Cross-Encoder	Joint	Joint	Full attention

Cross-encoders see the query and document together, allowing them to capture fine-grained relevance that bi-encoders miss.

Technique 2: LLM-Based Re-Ranking

Use an LLM to score and rank documents:

class LLMReranker:
    def __init__(self, llm):
        self.llm = llm
    
    async def rerank(self, query: str, documents: list[str], top_n: int = 5):
        """Re-rank documents using LLM scoring."""
        scoring_prompt = f"""Rate the relevance of each document to the query.
Return a JSON array with "index" and "score" fields (0-10).

Query: {query}

Documents:
{chr(10).join(f"{i}. {doc}" for i, doc in enumerate(documents))}

Scores:"""
        
        response = await self.llm.complete(scoring_prompt)
        
        # Parse scores
        import json
        scores = json.loads(response.text)
        
        # Rank by score
        ranked = sorted(
            enumerate(documents),
            key=lambda x: next(
                (s["score"] for s in scores if s["index"] == x[0]),
                0
            ),
            reverse=True
        )
        
        return [(doc, score) for idx, doc in ranked[:top_n] 
                for score in [next(
                    (s["score"] for s in scores if s["index"] == idx), 0
                )]]

LLM Re-Ranking Prompt Variants

PROMPTS = {
    "relevance": """Score how relevant this document is to answering the query.
Score 0-10 where 10 is perfectly relevant.""",
    
    "usefulness": """Would this document help answer the query?
Score 0-10 where 10 is extremely helpful.""",
    
    "extraction": """Can specific information be extracted from this document 
to answer the query? Score 0-10 where 10 means the answer is directly stated.""",
    
    "reasoning": """Think step by step about how this document relates to the query.
Then provide a relevance score 0-10."""
}

Technique 3: Reciprocal Rank Fusion (RRF)

Combine multiple ranking signals:

def reciprocal_rank_fusion(
    rankings: list[list[str]], 
    k: int = 60
) -> list[str]:
    """Combine multiple rankings using RRF."""
    fused_scores = {}
    
    for ranking in rankings:
        for rank, doc in enumerate(ranking):
            if doc not in fused_scores:
                fused_scores[doc] = 0
            # RRF formula: 1 / (k + rank)
            fused_scores[doc] += 1 / (k + rank + 1)
    
    # Sort by fused score
    ranked = sorted(
        fused_scores.items(),
        key=lambda x: x[1],
        reverse=True
    )
    
    return [doc for doc, score in ranked]

# Example: Combine vector search and BM25 rankings
vector_ranking = ["doc_a", "doc_b", "doc_c", "doc_d"]
bm25_ranking = ["doc_b", "doc_c", "doc_e", "doc_a"]

fused = reciprocal_rank_fusion([vector_ranking, bm25_ranking])
# ["doc_b", "doc_c", "doc_a", "doc_d", "doc_e"]

Technique 4: Diversity-Aware Re-Ranking

Remove near-duplicates and ensure coverage:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

class DiversityReranker:
    def __init__(self, embed_model, threshold: float = 0.85):
        self.embed_model = embed_model
        self.threshold = threshold
    
    async def rerank_with_diversity(
        self, 
        documents: list[str], 
        scores: list[float],
        top_n: int = 5
    ):
        """Re-rank with diversity constraint."""
        # Embed all documents
        embeddings = await self.embed_model.aget_text_embedding_batch(
            documents
        )
        embeddings = np.array(embeddings)
        
        # MMR (Maximal Marginal Relevance) selection
        selected = []
        remaining = list(range(len(documents)))
        
        # Select first document (highest score)
        first = max(remaining, key=lambda i: scores[i])
        selected.append(first)
        remaining.remove(first)
        
        while len(selected) < top_n and remaining:
            best_mmr = -float("inf")
            best_idx = None
            
            for idx in remaining:
                # Relevance score
                relevance = scores[idx]
                
                # Max similarity to already selected
                max_sim = max(
                    cosine_similarity(
                        embeddings[idx].reshape(1, -1),
                        embeddings[s].reshape(1, -1)
                    )[0][0]
                    for s in selected
                )
                
                # MMR = λ * relevance - (1-λ) * max_similarity
                lam = 0.7  # Balance relevance vs diversity
                mmr = lam * relevance - (1 - lam) * max_sim
                
                if mmr > best_mmr:
                    best_mmr = mmr
                    best_idx = idx
            
            selected.append(best_idx)
            remaining.remove(best_idx)
        
        return [documents[i] for i in selected]

Technique 5: Contextual Compression

Extract only the relevant portions from retrieved documents:

from llama_index.core.node_postprocessor import SentenceTransformerRerank
from llama_index.core.extractors import SummaryExtractor

class ContextualCompressor:
    def __init__(self, llm, reranker):
        self.llm = llm
        self.reranker = reranker
    
    async def compress(
        self, 
        query: str, 
        documents: list[str],
        max_tokens: int = 1500
    ):
        """Extract relevant sentences from documents."""
        compressed = []
        current_tokens = 0
        
        for doc in documents:
            # Split into sentences
            sentences = doc.split(". ")
            
            # Score each sentence
            scored_sentences = []
            for sentence in sentences:
                if sentence.strip():
                    score = self.reranker.predict(
                        [(query, sentence)]
                    )[0]
                    scored_sentences.append((sentence, score))
            
            # Take top sentences from each document
            scored_sentences.sort(key=lambda x: x[1], reverse=True)
            
            for sentence, score in scored_sentences:
                # Rough token estimate
                tokens = len(sentence.split())
                if current_tokens + tokens > max_tokens:
                    break
                compressed.append(sentence)
                current_tokens += tokens
        
        return ". ".join(compressed)

Putting It All Together: Complete Re-Ranking Pipeline

class RerankingPipeline:
    def __init__(self, retriever, cross_encoder, llm):
        self.retriever = retriever
        self.cross_encoder = cross_encoder
        self.llm = llm
    
    async def retrieve_and_rerank(
        self, 
        query: str,
        initial_k: int = 20,
        final_n: int = 5
    ) -> dict:
        """Complete re-ranking pipeline."""
        
        # Step 1: Initial retrieval
        initial_results = await self.retriever.aretrieve(query)
        documents = [r.node.text for r in initial_results]
        
        # Step 2: Cross-encoder re-ranking
        pairs = [(query, doc) for doc in documents]
        scores = self.cross_encoder.predict(pairs)
        
        # Step 3: Diversity filtering
        diversity_reranker = DiversityReranker(
            self.retriever._embed_model
        )
        diverse_docs = await diversity_reranker.rerank_with_diversity(
            documents, scores.tolist(), top_n=final_n * 2
        )
        
        # Step 4: Contextual compression
        compressor = ContextualCompressor(self.llm, self.cross_encoder)
        compressed = await compressor.compress(
            query, diverse_docs, max_tokens=1500
        )
        
        return {
            "compressed_context": compressed,
            "source_documents": diverse_docs,
            "original_count": len(documents),
            "final_count": len(diverse_docs),
        }

# Usage
pipeline = RerankingPipeline(retriever, cross_encoder, llm)
result = await pipeline.retrieve_and_rerank("how to optimize database queries")

Evaluation

Measure re-ranking impact:

async def evaluate_reranking(test_cases, pipeline):
    """Evaluate re-ranking quality."""
    metrics = {
        "mrr": [],      # Mean Reciprocal Rank
        "ndcg": [],     # Normalized Discounted Cumulative Gain
        "precision": [], # Precision at N
        "recall": [],    # Recall at N
    }
    
    for query, relevant_docs in test_cases:
        # Retrieve without re-ranking
        initial = await pipeline.retriever.aretrieve(query)
        initial_docs = [r.node.text for r in initial[:5]]
        
        # Retrieve with re-ranking
        reranked = await pipeline.retrieve_and_rerank(query)
        reranked_docs = reranked["source_documents"]
        
        # Calculate metrics
        metrics["mrr"].append(calculate_mrr(reranked_docs, relevant_docs))
        metrics["precision"].append(
            len(set(reranked_docs) & set(relevant_docs)) / len(reranked_docs)
        )
        metrics["recall"].append(
            len(set(reranked_docs) & set(relevant_docs)) / len(relevant_docs)
        )
    
    return {k: sum(v)/len(v) for k, v in metrics.items()}

Performance Considerations

Method	Latency	Accuracy	Cost
No re-ranking	0ms	Baseline	None
Cross-encoder	50-200ms	+15-25%	Low
LLM re-ranking	500-2000ms	+20-35%	Medium
RRF fusion	10-50ms	+10-20%	None
Diversity filtering	20-100ms	+5-15%	Low

Recommendation: Start with cross-encoder re-ranking. It provides the best accuracy-to-latency ratio. Add LLM re-ranking only for high-value queries.

Key Takeaways

1. Retrieval is not enough: Re-ranking dramatically improves precision
2. Cross-encoders are the sweet spot: Best accuracy per millisecond
3. Diversity matters: Remove near-duplicates to maximize context window value
4. Contextual compression reduces noise: Send only relevant sentences to the LLM
5. Combine techniques: RRF + cross-encoder + diversity is a powerful stack

Re-ranking transforms RAG from “retrieve everything, hope the LLM figures it out” to “retrieve precisely what the LLM needs.” The techniques here are battle-tested in production systems and will immediately improve your RAG quality.

References:

• LlamaIndex Node Postprocessors
• Sentence-Transformers Cross-Encoders
• Reciprocal Rank Fusion Paper