How to evaluate a RAG system: retrieval metrics, answer faithfulness and a test set

Key points

• Separate the halves: a wrong answer can come from a missing chunk or from a model that ignored the right one, and the fixes are different.
• The test set is the work: real questions, the ids of the chunks that answer them, reference answers, and some questions the documents cannot answer.
• Recall@k tells you whether the right chunk reached the model; MRR and nDCG tell you how high it ranked.
• Faithfulness asks whether each claim in the answer is supported by the retrieved context, which is not the same as being correct.
• LLM judges have known biases (position, verbosity, self-preference). Calibrate them against human labels before trusting a score.
• Gate CI on the cheap, deterministic retrieval metrics; track the noisier generation metrics against a baseline.

Evaluate retrieval and generation separately

A RAG system retrieves chunks, then asks a model to answer from them (see RAG explained). When the answer is wrong, the cause sits in one of two places, and an end-to-end "was the answer good" score cannot tell you which. So measure each stage on its own, with the same test set.

The general case for evals, success criteria and graders is in how to evaluate AI agents. This post covers what is specific to retrieval-augmented systems.

Build the test set

Every metric below needs labeled data. Each test case holds a question, the ids of the chunks or documents that answer it, and, for generation metrics that compare against a reference, a short reference answer.

1. Collect real questions first: search logs, support tickets, chat transcripts, questions sales or support staff hear often. Remove personal data before they enter the set.
2. Label relevant passages by a stable id. Labels tied to chunk ids break when you rechunk, so store the source document and a character span or quote as well, and map them to the current chunks when you rebuild the index.
3. Write a short reference answer for each question, from the source, not from the model.
4. Add questions the corpus cannot answer, with an empty relevant set. The right output there is "I don’t know", and you need to measure it.
5. Tag each case: question type (lookup, comparison, multi-hop, identifier), topic, and source (real or synthetic). Report metrics per tag as well as overall.
6. Keep a held-out part you do not look at while tuning, so you do not overfit the prompt to the test set.

Synthetic generation helps fill gaps: ask a model to write questions from a chunk, and that chunk becomes the label. Use it with care. Questions written while looking at a chunk tend to reuse its exact words, which flatters keyword retrieval and makes the set easier than real traffic. They also miss questions that need two documents, and questions whose answer is not in the corpus at all. Have a person review generated questions, discard the trivial ones, keep them tagged as synthetic, and never let them replace real questions.

Size matters less than coverage. Anthropic’s eval guidance favors more questions with slightly noisier automated grading over a few hand-graded ones. Start with what you can label well, make sure every question type and every important document group is present, and grow the set from real failures.

Retrieval metrics: recall@k, precision@k, MRR and nDCG

These are the standard ranked-retrieval measures from information retrieval. Each is computed per question and averaged over the set.

For one query q, with ranked results R(q) and labeled relevant set Rel(q):

precision@k = (relevant results in the top k) / k
recall@k    = (relevant results in the top k) / |Rel(q)|
RR(q)       = 1 / (rank of the first relevant result), or 0 if none
MRR         = mean of RR(q) over all queries

DCG@k  = sum for i = 1..k of  rel_i / log2(i + 1)
nDCG@k = DCG@k / IDCG@k
  rel_i  = graded relevance of the result at rank i, for example 0, 1 or 2
  IDCG@k = DCG@k of the ideal ordering of all labeled relevant items

nDCG comes from Järvelin and Kekäläinen’s work on cumulated gain. Their original discount applies log_b(i) from rank b onward; the log2(i + 1) form above is the common variant, and some tools also use an exponential gain. Report which one you use. A frequent bug is computing the ideal ordering from the retrieved list instead of from all labeled relevant items, which inflates the score when a relevant chunk was never retrieved.

Recall@k and MRR in plain Python. Unanswerable questions have no relevant set, so they are skipped here and scored on the answer side instead.

def recall_at_k(retrieved, relevant, k):
    """retrieved: ranked ids, best first. relevant: set of labeled ids."""
    if not relevant:
        return None  # unanswerable question: score it separately
    return len(set(retrieved[:k]) & relevant) / len(relevant)

def reciprocal_rank(retrieved, relevant):
    for rank, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0

def evaluate(run, labels, k=5):
    recalls, rrs = [], []
    for qid, relevant in labels.items():
        retrieved = run.get(qid, [])
        r = recall_at_k(retrieved, relevant, k)
        if r is not None:
            recalls.append(r)
            rrs.append(reciprocal_rank(retrieved, relevant))
    return {"recall@k": sum(recalls) / len(recalls), "mrr": sum(rrs) / len(rrs)}

labels = {"q1": {"c4"}, "q2": {"c2", "c9"}, "q3": set()}
run = {"q1": ["c1", "c4", "c7"], "q2": ["c9", "c3", "c5"], "q3": ["c8"]}
print(evaluate(run, labels, k=3))  # recall@k 0.75, mrr 0.75

Retrieval metrics are deterministic and cheap: no model call, just ids. That makes them the right thing to compare chunking strategies (chunking strategies for RAG), embedding models, and keyword versus hybrid search.

Generation metrics: faithfulness, answer relevance and context precision

The RAGAS paper (Es, James, Espinosa-Anke and Schockaert) defines three reference-free metrics, computed with an LLM rather than with human labels.

Faithfulness       F  = |V| / |S|
  S = statements an LLM extracts from the answer
  V = statements the LLM judges supported by the retrieved context

Answer relevance   AR = (1/n) * sum for i = 1..n of cos( E(q), E(q_i) )
  q_i = questions an LLM generates from the answer alone
  E   = an embedding model

Context relevance  CR = (sentences extracted as needed) / (sentences in the context)

• Faithfulness: is every claim in the answer supported by the retrieved context? It is the closest metric to "did it hallucinate". A faithful answer can still be wrong if the retrieved context was wrong, so pair it with retrieval metrics.
• Answer relevance: does the answer address the question? The paper’s trick is to generate questions from the answer and compare them with the original; an incomplete or padded answer generates questions that drift. It does not check facts.
• Context relevance or precision: is the retrieved context focused on the question? The current Ragas library defines context precision differently from the paper: it averages precision@k at the ranks where a relevant chunk appears, with relevance judged against a reference answer or the response, so it rewards relevant chunks ranked near the top.

Library definitions change between versions, so record the version and metric names with every result. Where you have reference answers, add an answer-correctness check against them: faithfulness tells you the model stayed within the context, correctness tells you the result was right.

You can compute faithfulness without a framework. A judge prompt along these lines works, with the score computed in code from the judge’s labeled list rather than taken from the judge’s own arithmetic.

You are checking whether an answer is supported by a context.

<context>
{retrieved_chunks}
</context>

<answer>
{answer}
</answer>

1. List each factual claim in the answer, one per line.
2. For each claim, write SUPPORTED or NOT_SUPPORTED and quote the
   context sentence that supports it, if any.
3. A claim that is true but not stated in the context is NOT_SUPPORTED.

Return JSON: {"claims": [{"claim": "...", "verdict": "...", "quote": "..."}]}

LLM-as-judge and its biases

Most generation metrics use a model as the grader. Zheng et al., in "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", report that strong judges reached over 80% agreement with human preferences, the same level humans reach with each other. They also document the biases you have to design around.

Before trusting a judge, measure it. Have people label a sample of answers with the same rubric, run the judge on that sample, and compute agreement. If agreement is low, fix the rubric before you look at the scores. Prefer narrow, binary questions ("is this claim in the context?") over open 1-to-10 ratings; they are easier to calibrate and less noisy. Re-check agreement whenever you change the judge model or prompt.

Regression testing in CI

A RAG system changes in many places: the chunker, the embedding model, the index settings, the fusion weights, the prompt, the model. Any of them can fix one question and break three others. Treat the test set like a test suite and run it on every change (what CI/CD is).

1. Version the test set and the index build together, so a score always refers to a known corpus, chunking and embedding model.
2. Run retrieval metrics on every pull request that touches retrieval. They are fast and deterministic, so fail the build when recall@k drops below the baseline.
3. Run generation metrics on changes to prompts, models or retrieval. Judge scores are noisy, so compare against the stored baseline with a tolerance, and look at which questions changed rather than only the average.
4. Store per-question results, not just totals, so a reviewer can open the five questions that regressed.
5. Keep a small set of must-pass cases, such as refusals on unanswerable questions and known past failures, that fail the build on any regression.

Pin the judge model and its settings, and rerun the baseline when you change them; otherwise a score change may come from the grader, not the system.

Online signals after launch

Offline metrics only cover the questions you thought of. After launch, real traffic shows what is missing. None of these signals is a metric on its own; they point you to conversations worth reviewing.

• Explicit feedback: thumbs up or down, and the reason when the user gives one.
• Behavior: the user rephrases the same question, opens the cited source, copies the answer, or asks for a person.
• No-answer rate: how often the system says it does not know. A sudden rise can mean an index problem; a fall to zero can mean it has started guessing.
• Retrieval health: queries whose top retrieval score is unusually low, and queries that return nothing from the keyword side.
• Sampled review: people grading a random sample of real conversations with the same rubric as the offline judge, which also keeps the judge honest.

Close the loop: every confirmed failure becomes a test case, with its relevant chunks labeled, so it cannot regress silently. Most failures you find this way trace back to retrieval or to answers that went beyond the context; how to reduce AI hallucinations covers the fixes on the generation side, and RAG vs fine-tuning covers when retrieval is the wrong tool.

BYD Tech builds AI agents and assistants with their evals: the code lives in the client’s GitHub from day one, and AI-assisted work is reviewed by an engineer.