RAG Step 4: Generating Grounded Answers with Citations

Step 1: Label each retrieved chunk

Number the sources and include where each came from. Numbered labels let the model cite with short markers, and your app can turn those markers back into file names and pages.

def build_prompt(question: str, hits: list[dict]) -> str:
    sources = []
    for n, hit in enumerate(hits, start=1):
        label = hit["source"]
        if hit.get("page"):
            label += ", page " + str(hit["page"])
        sources.append("[" + str(n) + "] (" + label + ")\n" + hit["text"])
    context = "\n\n".join(sources)
    return "<sources>\n" + context + "\n</sources>\n\nQuestion: " + question

Step 2: Write grounding instructions

Tell the model to use only the sources, cite them like [1], and say plainly when the sources do not contain the answer. An explicit refusal path is the best defense against made-up answers.

SYSTEM = (
    "You answer questions using only the numbered sources provided. "
    "Cite the sources you use like [1] or [2]. "
    "If the sources do not contain the answer, say: "
    "'I don't have enough information in the documents to answer that.' "
    "Do not use outside knowledge."
)

Step 3: Generate the answer

Combine retrieval and generation in one function. If retrieval returns nothing above the threshold, skip the model call and return the fallback message, which saves cost and avoids a speculative answer.

import anthropic

client = anthropic.Anthropic()

def answer(question: str, store: "VectorStore") -> dict:
    hits = store.search(question, k=4)
    if not hits:
        return {"answer": "I don't have enough information in the documents to answer that.", "sources": []}

    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=16000,
        system=SYSTEM,
        messages=[{"role": "user", "content": build_prompt(question, hits)}],
    )
    text = next(b.text for b in response.content if b.type == "text")
    return {"answer": text, "sources": hits}

Step 4: Show citations to the user

Display the answer with the sources behind it so users can verify claims. Check that every citation marker in the answer maps to a source you actually provided, and flag answers with no citations for review.

result = answer("How many vacation days do new employees get?", store)
print(result["answer"])
for n, hit in enumerate(result["sources"], start=1):
    print("[" + str(n) + "]", hit["source"], hit.get("page"))

Step 5: Consider built-in citation features

Some APIs, including Claude's, can attach citations to source documents natively when you pass documents as content blocks, returning exact cited passages. That can replace hand-built numbering, though it has its own constraints, such as not combining with structured output, so check the documentation for your model.