Building a Production-Ready RAG Pipeline with LlamaIndex

A toy RAG app — load a folder, build a vector index, ask a question — works in a notebook and falls apart in production. Real RAG is a pipeline with distinct, individually tunable stages: load → chunk → embed → index → retrieve → post-process → synthesize → evaluate. Each stage has failure modes that the “20-line demo” hides.

In this tutorial, you will learn how to build a production-shaped RAG pipeline with LlamaIndex: deterministic chunking with metadata, persistent storage, hybrid retrieval with reranking, and evaluation with a labelled dataset.

Prerequisites

• Python 3.10+, pip install llama-index-core llama-index-llms-gemini llama-index-embeddings-gemini
• A Gemini API key (or swap in any LlamaIndex LLM/embed_model)

Configure a single Settings object

LlamaIndex centralizes model configuration on a global Settings object, so every component shares the same LLM and embedder:

from llama_index.core import Settings
from llama_index.llms.gemini import Gemini
from llama_index.embeddings.gemini import GeminiEmbedding

Settings.llm = Gemini(model="models/gemini-2.5-pro", api_key="...")
Settings.embed_model = GeminiEmbedding(
    model="models/text-embedding-004", api_key="..."
)
Settings.chunk_size = 512
Settings.chunk_overlap = 64

Ingestion: chunk with metadata, not just text

The single biggest quality lever in RAG is how you split documents. Naive character splits shred sentences and lose context. Use SentenceSplitter and attach metadata (source file, section) so retrieval can be filtered and cited:

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.extractors import TitleExtractor
from llama_index.core.readers import SimpleDirectoryReader

reader = SimpleDirectoryReader("data/", filename_as_id=True)
docs = reader.load_data()

pipeline = IngestionPipeline(
    transformations=[
        SentenceSplitter(
            chunk_size=Settings.chunk_size,
            chunk_overlap=Settings.chunk_overlap,
        ),
        TitleExtractor(metadata_name="title", llm=Settings.llm),
        Settings.embed_model,
    ],
)
nodes = pipeline.run(documents=docs)

TitleExtractor enriches every node with a title — cheap, and it measurably improves retrieval for long documents.

Index: persist it, don’t rebuild it

A production index lives in a real vector store, not in memory. Persist to disk (or PostgreSQL via pgvector) so a restart doesn’t re-embed thousands of chunks:

from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.storage.docstore import SimpleDocumentStore
from llama_index.core.storage.index_store import SimpleIndexStore

storage = StorageContext.from_defaults(
    docstore=SimpleDocumentStore.from_documents(nodes),
    index_store=SimpleIndexStore.from_documents(nodes),
)
storage.docstore.add_documents(nodes)
index = VectorStoreIndex(nodes, storage_context=storage)
index.storage_context.persist("index_store")

Loading later is a single call: VectorStoreIndex.load_from_disk("index_store").

Retrieval: hybrid search + reranking

Dense vector search alone misses exact terms (IDs, code symbols, names). Combine it with a sparse BM25 retriever and fuse the results, then rerank with a cross-encoder. This hybrid pattern is the difference between “kinda relevant” and “the right paragraph”:

from llama_index.core.retrievers import QueryFusionRetriever, VectorIndexRetriever
from llama_index.core.retrievers import BM25Retriever
from llama_index.core.postprocessor import SentenceTransformerRerank

vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=8)
bm25_retriever = BM25Retriever.from_defaults(docstore=index.docstore, top_k=8)

retriever = QueryFusionRetriever(
    [vector_retriever, bm25_retriever],
    similarity_top_k=5,
    num_queries=1,
    mode="reciprocal_rerank",
)

reranker = SentenceTransformerRerank(model="BAAI/bge-reranker-base", top_n=3)

Query: full query engine with citations

With retrieval solved, build the query engine and add a node postprocessor that attaches source metadata to the response so users can verify the answer:

from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.response_synthesizers import get_response_synthesizer

synthesizer = get_response_synthesizer(response_mode="compact")
query_engine = RetrieverQueryEngine(
    retriever=retriever,
    response_synthesizer=synthesizer,
    node_postprocessors=[reranker],
)

response = query_engine.query("How does chunking affect retrieval quality?")
for node in response.source_nodes:
    print(node.node.metadata.get("file_name"), "->", node.score)

Evaluation: measure before you trust it

Production RAG is a measured system. Build a small labelled set of (question, expected answer) pairs and score retrieval and response quality:

from llama_index.core.evaluation import (
    RetrieverEvaluator, RelevancyEvaluator, generate_question_context_pairs,
)
from llama_index.core import Document

qa_pairs = generate_question_context_pairs(nodes, llm=Settings.llm, num_questions_per_chunk=1)

retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["hit_rate", "mrr"], retriever=retriever
)
results = await retriever_evaluator.aevaluate_dataset(qa_pairs)
print({k: round(v, 3) for k, v in results.metric_results_dict.items()})

Track hit_rate and MRR on every change. When you tune chunk size or reranker, you’ll know if it actually helped.

Putting It All Together

The complete, runnable pipeline — ingestion, persistent index, hybrid retrieval, reranking, and evaluation — is in this gist. Run the eval script first to get a baseline, then change one variable (chunk size, top_k) and watch the metric move.

Conclusion & Next Steps

You now have a production RAG pipeline with tunable stages and objective evaluation. Next: add an IngestionPipeline that only re-embeds changed documents, plug in PostgreSQL with pgvector for shared storage, and automate the eval run in CI so a prompt change can’t silently degrade retrieval.

References / Sources

• LlamaIndex high-level concepts (LLMs, agents, RAG). https://docs.llamaindex.ai
• LlamaIndex cost analysis and Settings configuration. https://docs.llamaindex.ai/en/stable/understanding/evaluating/cost_analysis/
• LlamaIndex evaluation module guides. https://docs.llamaindex.ai/en/stable/module_guides/evaluating/