Grounded Agents: Citation-Anchored Answers in RAG

“According to the documentation…” is not a citation. It’s a vibe. Users — and the reviewers who sign off on what your users ship — want to click something and land on the exact passage that justifies a claim. That’s the difference between a RAG demo and a RAG system people rely on.

Citation-anchored answering isn’t a model feature you turn on. It’s a contract you enforce on three sides: the retrieval side (emit stable, addressable chunk identity), the generation side (a prompt contract with verifiable structure), and the rendering side (provenance the user can act on). This post walks the whole loop.

Citations are data, not prose

The core design decision: a citation is a structured reference to a retrieved object, not a sentence the model wrote. If your citation lives only inside the answer string, you can’t validate it, filter it for access control, or link it anywhere.

So give every chunk an identity before it reaches the model, and keep the mapping on your side of the wire:

from dataclasses import dataclass, field

@dataclass
class RetrievedChunk:
    chunk_id: str          # unique within this retrieval, e.g. "c3"
    doc_id: str            # stable across queries, e.g. "handbook#v12"
    title: str             # human label: doc title + heading path
    url: str               # deep link: doc URL + #heading-anchor
    text: str              # exact text the model will see
    score: float = 0.0
    span: tuple[int, int] | None = None  # offsets inside the source doc
    meta: dict = field(default_factory=dict)

Two IDs, deliberately different. chunk_id is a per-query handle the model can cite cheaply (c3). doc_id + span is durable provenance you can resolve later, after the conversation, when someone asks “where did this come from?” A chunk ID that’s only unique inside one retrieval session is useless for audit; a doc ID that changes whenever you re-chunk loses history. Keep both.

The prompt contract

Retrieval quality buys you nothing if the model treats the context as inspiration. Pin down the rules explicitly and make the failure mode answerable:

You are answering strictly from the numbered passages provided.

RULES:
1. Every factual claim must carry a citation marker like [c1] or [c2][c5].
2. Cite only passage IDs that appear in PASSENGERS below. Never invent an ID.
3. Quotes must be verbatim substrings of the cited passage.
4. If the passages don't contain the answer, reply exactly:
   "Not found in the provided sources." — do not guess.
5. Do not cite a passage for a claim it does not contain.

PASSENGERS:
[c1] (Handbook > Time off) ...
[c2] (Handbook > Expenses) ...

Rules 3 and 4 are the load-bearing ones. Rule 3 turns a citation into something checkable; rule 4 gives the model a socially acceptable way out, which measurably reduces confabulation compared with “always answer the question.” In agent loops — the kind of thing smolagents and friends run, where a retrieval tool gets called mid-trajectory — apply the same contract per tool result, not just to the final answer. A tool call that returns passages without IDs gives the model nothing to anchor to.

Retrieve with attribution in mind

Grounding quality is mostly set before generation:

• Chunk with provenance. Store heading path, page number, or line range at split time. Recovering offsets after the fact is guesswork.
• Rerank, then renumber. If you rerank (Sentence Transformers’ reranking models or a dedicated reranker are the usual picks on the Hugging Face docs index), renumber the citation IDs after ranking so c1 is always the top hit. IDs that don’t follow rank order make answers look arbitrary.
• Deduplicate aggressively. Near-identical chunks from overlapping windows produce contradictory citations for the same claim. Users notice.
• Keep the retrieval result in session state. The model cited c3; you must be able to resolve c3 back to text in that same request. Re-running retrieval to resolve a citation will often return a different set.

Verify before you render

Post-generation validation is what makes the citation trustworthy rather than decorative:

def validate(answer: str, chunks: list[RetrievedChunk]) -> tuple[str, list[str]]:
    by_id = {c.chunk_id: c for c in chunks}
    cited = re.findall(r"\[(c\d+)\]", answer)
    unknown = sorted({i for i in cited if i not in by_id})

    if unknown:
        # a cited passage that was never retrieved = fabrication
        raise ValueError(f"citations to unknown passages: {unknown}")

    for quote in re.findall(r'"([^"]{20,})"', answer):
        norm = lambda s: re.sub(r"\s+", " ", s).strip().lower()
        if not any(norm(quote) in norm(c.text) for c in chunks):
            raise ValueError(f"quote not found in any passage: {quote[:60]}")

    return answer, sorted(set(cited))

On failure, don’t drop the answer silently. Retry once with the validator’s error appended to the prompt (“your citation [c9] does not exist”), then degrade to the unverified answer with a visible “sources not verified” flag. A visible downgrade is honest; a quietly stripped citation is worse than none, because the user assumes the rest passed too.

Rendering provenance users can act on

Anchoring is only real if the UI makes it one click deep:

• Inline marker → hover card. [c2] opens a card with the doc title, heading path, the quoted span highlighted, and a “Open source” link to url with the #anchor.
• Side panel as the source of truth. List every cited chunk once, in citation order, with scores hidden (scores confuse users) and the exact quoted text shown.
• Highlight on navigate. Landing on url should scroll to and flash the span — otherwise “verification” means re-reading a 40-page document, which nobody does.
• Filter by access control. Because citations are structured, you can drop chunks the current user isn’t allowed to see before rendering. String-only citations make this impossible, and the leak vector is real in multi-tenant corpora.
• Show nothing when there’s nothing. An answer with zero validated citations should render differently from one with three. Differentiate them; don’t normalize both into the same confident-looking card.

Measure it

You can’t improve what you don’t score. Three cheap metrics over a small hand-checked eval set (store it as a Hugging Face Dataset so runs are comparable):

1. Citation precision — of cited passages, fraction that actually support the claim (sample and human-grade 50 answers; this is the honest number).
2. Quote correctness — fraction of verbatim quotes that pass your substring validator. Purely automated, catches model drift immediately.
3. Attribution coverage — fraction of factual sentences carrying at least one citation. Automated, catches the model getting lazy.

Run quote correctness in CI on a fixed prompt + fixed retrieval set. When it drops, either the model changed, your prompt drifted, or your chunker regressed — all three are worth knowing before your users do.

Pitfalls and gotchas

• Paraphrase presented as a quote. Models love “polishing” quotes. Normalize whitespace/case when validating, but reject anything that isn’t a substring — a close-but-wrong quote is a hallucination with good formatting.
• Markers drift under streaming. If you render markers as they arrive and the model later rewrites them, you get orphaned anchors. Buffer citations and resolve after the stream completes.
• Citations pointing at the wrong granularity. Citing a whole document for a sentence-level claim passes validation but fails the user. Push toward the smallest chunk that supports the claim.
• Multi-chunk claims. “X and Y” backed by [c1][c7] may mean each half is supported — or that the model cross-wired them. When stakes are high, validate per-sentence, not per-answer.
• Retrieval truncation. Long contexts get cut; if the PASSENGERS block is truncated mid-chunk you’ll see phantom IDs from the model’s imagination. Truncate at chunk boundaries and re-check that every ID in the prompt still resolves.
• Access-controlled chunks in the prompt. Don’t retrieve across tenants and filter at render time — filter before generation, or the model will quote things the user can’t see and the citation will dead-end.
• Don’t fabricate anchors. If you don’t have a heading anchor for a PDF page, link to the file and say “page 12”. An anchor that lands in the wrong place destroys more trust than no anchor.

Sources

• Hugging Face documentation index — Hub, Sentence Transformers, Datasets, and agent libraries
• smolagents documentation — agent loops and tool-calling agents
• Sentence Transformers — embeddings and reranking