RAG Evaluation: Metrics, Testing & Hallucination Checks

Retrieval-Augmented Generation (RAG) can fail in more than one way. The retriever may return irrelevant passages, the generator may ignore useful context, or the final answer may sound convincing while making claims that the documents do not support. A few manual spot checks cannot tell you which failure is happening.

This guide gives small teams a practical evaluation method that does not require a large platform. It explains the metrics, shows how to build a compact test set, and proposes an experiment that compares a baseline retriever with one controlled improvement. It also explains what the numbers cannot prove.

What RAG evaluation actually measures

RAG evaluation is not one score. It is a set of checks across the path from a user question to a generated answer. The most useful split is between retrieval, grounding, and answer quality.

These layers diagnose different problems. A correct answer with irrelevant retrieved passages may be lucky. A relevant passage with an unsupported answer points to a generation or prompt problem. A low score can therefore be useful only when you know which layer produced it.

Build a small test set before choosing metrics

A test set is a collection of representative questions that you can run repeatedly. It does not need to be large to be useful. A small, carefully designed set is better than hundreds of easy questions that never expose failure modes.

For a first evaluation, create 20 questions from the documents your system is meant to answer. Keep the source documents and expected evidence alongside each question. Include several types:

• Single-document questions: the answer is stated clearly in one source.
• Multi-document questions: the answer requires combining evidence from two or more sources.
• Unanswerable questions: the information is absent and the correct behavior is to say so.
• Ambiguous questions: the wording has more than one plausible interpretation.
• Adversarial or noisy questions: the prompt contains distracting terms or asks for a conclusion that the evidence does not support.

For each case, record the question, expected answer or key facts, relevant document IDs, acceptable alternative answers, and the failure type you want to detect. Do not silently rewrite a question after seeing a poor result. Otherwise the test set becomes a moving target.

Understand the core RAG metrics

Precision@k and recall@k

Precision@k asks how many of the top-k retrieved passages are relevant. Recall@k asks whether the passages needed for the answer appeared in the retrieved set. Precision helps identify noisy context; recall helps identify missing evidence. Both require a relevance judgment or a known set of expected documents.

Mean reciprocal rank

Mean reciprocal rank (MRR) rewards a system when the first relevant result appears near the top. It is useful when the generator mainly relies on early context, but it does not tell you whether all necessary evidence was retrieved.

Faithfulness or groundedness

Faithfulness asks whether the answer is supported by the retrieved context. It is different from correctness. An answer can be factually correct because the model already knew it, while still failing to use or cite the supplied evidence. For a knowledge-base assistant, unsupported certainty is often a more important failure than awkward wording.

Answer relevance and correctness

Answer relevance measures whether the response addresses the question without unnecessary detours. Correctness compares the response with a reference answer or a set of expected facts. Use reference answers carefully: they should describe acceptable content, not force one exact wording.

Latency and cost

A system that gives accurate answers but takes too long or costs too much may still fail its intended use case. Record retrieval latency, generation latency, token usage, and error rate alongside quality metrics. These are operational constraints, not substitutes for factual evaluation.

A repeatable RAG evaluation workflow

1. Freeze the question set. Version the questions, expected evidence, and scoring rules.
2. Run a baseline. Record the retriever, embedding model, chunk size, top-k value, prompt, model, and date.
3. Inspect failures by layer. Decide whether the issue came from missing retrieval, irrelevant context, unsupported generation, or an unclear question.
4. Change one meaningful variable. For example, change chunking, add a reranker, adjust top-k, or revise the grounding instruction. Do not change all of them at once.
5. Run the same test set again. Compare the new results with the baseline and preserve failures that did not improve.
6. Set a release rule. Block a release if critical cases become ungrounded, if the abstention behavior regresses, or if latency exceeds the product limit.

Practical experiment: baseline versus one controlled change

Use a small corpus of 8–12 documents and a 20-question test set. Start with dense retrieval and record the top-k passages for every question. Then choose one improvement based on the observed failure pattern.

If relevant passages are missing, test a different chunking strategy or hybrid retrieval. If the right passages are present but answers are unsupported, test a stricter grounding instruction and require evidence-linked responses. If the top results contain duplicates or near-matches, test reranking or metadata filters.

for case in test_set:
    retrieved = rag.retrieve(case.question, top_k=5)
    answer = rag.generate(case.question, retrieved)

    scores = {
        "retrieval_relevance": score_retrieval(retrieved, case.relevant_docs),
        "groundedness": score_grounding(answer, retrieved),
        "answer_relevance": score_relevance(answer, case.question),
        "correctness": score_correctness(answer, case.reference),
    }
    save_result(case.id, scores, retrieved, answer)

compare_with_baseline()
apply_release_gate()

The code is a testing pattern, not a drop-in library. The important design choice is that it saves the retrieved evidence and the answer together. Without that record, a low answer score is difficult to diagnose.

Do not treat an LLM judge as ground truth

LLM-based evaluators can scale review, but they can also inherit biases, miss subtle contradictions, and prefer fluent answers. Use them as one signal. For high-risk cases, add human review or deterministic checks such as required citations, exact identifiers, schema validation, or forbidden-claim tests.

A good evaluation report shows the evaluator prompt, model, version, rubric, and sample failures. If the judge changes, rerun a calibration set. Do not compare scores from different judges as if they were on one stable scale.

Common mistakes that make RAG evaluation misleading

• Testing only easy questions: add multi-hop, ambiguous, and unanswerable cases.
• Using one blended score: keep retrieval, grounding, answer quality, and operations separate.
• Changing several variables: you will not know what caused the improvement.
• Rewarding confident guesses: score abstention when evidence is missing.
• Ignoring the corpus: a metric result applies to the documents and questions tested.
• Promoting without a regression set: a change that helps one category can damage another.

Release checklist for a small RAG team

• The test set contains answerable, multi-document, ambiguous, and unanswerable questions.
• Relevant evidence is recorded for cases where retrieval quality is measured.
• Retrieval and generation scores are reported separately.
• Groundedness is checked against the actual retrieved context.
• Critical answers have a human or deterministic verification path.
• Every experiment records the model, prompt, embedding model, top-k, chunking, and date.
• A regression gate blocks releases that harm critical cases.
• Latency, cost, and error rate are reviewed with quality metrics.
• Evaluation results are not presented as universal benchmarks outside the tested corpus.

Frequently asked questions

What is the most important RAG metric?

There is no universal winner. Start with retrieval relevance, groundedness, and answer correctness. The right priority depends on whether your main failure is missing context, unsupported generation, or an incorrect response.

Can a RAG system be correct but not faithful?

Yes. A model may produce a correct answer from prior knowledge even when the retrieved documents do not support it. If your application promises document-grounded answers, faithfulness still matters.

How many questions should a first test set contain?

Twenty carefully selected questions are a reasonable starting point for a small experiment. Expand the set with real failures and new user intents as the system evolves.

Should I use Ragas or LangSmith?

Tools can simplify dataset management and scoring, but the evaluation design comes first. Choose a tool that fits your stack, supports the metrics you need, and lets you inspect failures rather than only displaying a single score.

Does a high RAG score prove production readiness?

No. It shows performance on the selected questions, documents, evaluator, and configuration. Production readiness also requires security, privacy, latency, cost, monitoring, and a plan for changing data.

Conclusion

Reliable RAG evaluation starts with a small, representative test set and a clear separation between retrieval, grounding, answer quality, and operations. Run a baseline, change one variable, preserve the evidence, and apply a release gate to the cases that matter. This process will not produce a universal ranking for your system, but it will tell you where the system fails and whether a change actually helped.

Editorial note: Evaluation frameworks and model behavior change over time. Recheck tool documentation and rerun your test set when you change models, retrieval settings, documents, or prompts.

Sources and further reading

1. LangSmith: Evaluate a RAG application — dataset creation, evaluation runs, correctness, relevance, groundedness, and retrieval relevance.
2. Qdrant: RAG evaluation guide — retrieval failures, chunking, embeddings, Precision@k, MRR, NDCG, and reranking.
3. Evaluation of Retrieval-Augmented Generation: A Survey — research taxonomy across retrieval, generation, and end-to-end evaluation.
4. Google Cloud: Optimizing RAG retrieval — testing frameworks and retrieval improvement workflows.
5. Ragas documentation — metrics and practical evaluation patterns.

Related PromptSphere reading: AI Agent Evaluation Dataset, AI Agent Evaluation Release Gate, AI Agent Observability, and LLM Model Routing.