RAG Evaluation: Measuring Retrieval Quality Objectively

A RAG pipeline has two halves that fail in different ways: the retriever (did it fetch the right chunks?) and the generator (did it answer faithfully from those chunks?). Most debugging stops at “the answer looks wrong,” which conflates the two. Objective evaluation separates them with distinct metrics, so you know which half to fix. — “AI does not create knowledge; it uses knowledge.” A RAG system’s worth is a direct function of how well it retrieves the right knowledge — and you cannot know that without measuring it.

This tutorial builds a small, self-contained eval harness in Python over a labeled set of (question, expected-source) pairs, computing retrieval metrics (hit_rate, MRR, context precision/recall) and generation metrics (faithfulness, answer_relevancy). The framework ideas follow LlamaIndex evaluation and Hugging Face practice; the math is small enough that the harness is transparent and auditable.

Prerequisites

• pip install llama-index-core (RAGAS optional).
• A vector store with your chunks and an embedding model. Metric definitions: hit_rate = was any relevant chunk in the top-k; MRR = reciprocal rank of the first relevant chunk; faithfulness = is the answer grounded in retrieved context.

Step 1: Scaffold a labeled eval dataset

EVAL = [
    {
        "query": "What is context caching and when does it help?",
        "gold_chunks": ["doc2.txt#12"],       # chunks that MUST be retrieved
        "expected_answer": "Caching reduces input cost for repeated context.",
    },
    # ... 30–100 rows. 100 is a common target for stable averages.
]

Step 2: Rank position of the first relevant chunk

Implement MRR and hit_rate against your actual retriever, given the ordered list of returned node_ids.

def mrr(retrieved_ids, gold_ids):
    for rank, r in enumerate(retrieved_ids, start=1):
        if r in gold_ids:
            return 1.0 / rank
    return 0.0

def hit_rate(retrieved_ids, gold_ids):
    return 1.0 if any(r in gold_ids for r in retrieved_ids) else 0.0

def eval_retrieval(retriever, dataset, top_k=5):
    hr, mrr_ = [], []
    for row in dataset:
        ids = [n.node_id for n in retriever.retrieve(row["q"])][:top_k]
        hr.append(hit_rate(ids, row["answer_chunks"]))
        mrr_.append(mrr(ids, row["answer_chunks"]))
    return {"hit_rate": sum(hr)/len(hr), "mrr": sum(mrr_)/len(mrr_)}

Low hit_rate/mrr means the retriever is missing the source from which the right answer must come — fix the embeddings/chunking, not the prompt.

Step 3: Context precision and recall

Where hit_rate/MRR look at a single relevant chunk, context precision and recall look at the whole retrieved set:

• Context precision — of the chunks retrieved for a query, what fraction was actually relevant?
• Context recall — of the chunks needed to answer, what fraction was retrieved?

These are the two sides of over-/under-retrieval and they trade off.

def context_precision(retrieved_ids, relevant_ids):
    if not retrieved_ids:
        return 0.0
    return len(set(retrieved_ids) & set(relevant_ids)) / len(retrieved_ids)

def context_recall(retrieved_ids, relevant_ids):
    if not relevant_ids:
        return 1.0
    return len(set(retrieved_ids) & set(relevant_ids)) / len(relevant_ids)

Step 4: Faithfulness and answer relevancy

Retrieval metrics never see the generated answer. Faithfulness checks the model didn’t ghost into territory the retrieved context doesn’t support (hallucination); answer relevancy checks the answer actually addresses the query.

A minimal judge version of faithfulness (RAGAS-style, statement-by-statement verification):

from pydantic import BaseModel

class Verdict(BaseModel):
    supported: bool

def faithfulness(answer: str, contexts: list[str]) -> float:
    ctx = "\n".join(contexts)
    stmts = split_statements(answer)                  # simple sentence split
    results = []
    for s in stmts:
        v = client.models.generate_content(
            model="gemini-2.5-flash",
            contents=(f"Context:\n{ctx}\n\nStatement:\n{s}\n"
                      f"Supported by the context? (true/false)"),
            config={"response_mime_type": "application/json", "response_schema": Verdict},
        )
        results.append(Verdict.model_validate_json(v.text).supported)
    return sum(results) / len(results)

Step 5: Full harness runs and reports

def evaluate(rag, queries):
    rows = []
    for row in queries:
        ids = [n.node_id for n in rag.retrieve(row["q"])]
        answer = rag.answer(row["q"])            # generates from retrieved contexts
        rows.append({
            **row,
            "hit_rate": hit_rate(ids, row["answer_chunks"]),
            "mrr": mrr(ids, row["answer_chunks"]),
            "context_precision": context_precision(ids, row["relevant"]),
            "context_recall": context_recall(ids, row["relevant"]),
            "faithfulness": faithfulness(answer, rag.contexts_for(row["q"])),
        })
    return pd.DataFrame(rows).mean(numeric_only=True)

Putting It All Together

Run it before and after any change — new embedding model, chunk size, reranker, prompt. If hit_rate stays flat but faithfulness drops, your model is the problem; if hit_rate falls, your retriever is. Version the dataset and the scores so “did RAG improve?” has a number, not a debate.

Conclusion & Next Steps

RAG evaluation turns “the answer was wrong” into an attributable metric on a specific component. Next: grow the eval set to 100+ and slice by query type, add a CI regression gate that fails a deploy when hit_rate/faithfulness drops, and track retrieved-but-unused chunks that inflate context and cost.

References / Sources

• LlamaIndex retrieval evaluation. https://docs.llamaindex.ai
• Hugging Face docs. https://huggingface.co/docs
• RAGAS metrics. https://docs.ragas.io