RAG evaluation is the process of testing whether a retrieval-augmented generation application finds the right information, uses it accurately, and communicates its answer with appropriate evidence. This reusable framework helps developers create test sets, measure retrieval quality, check faithfulness and citations, test prompts, and establish an evaluation workflow that can be repeated as data, models, and application code change.
Overview
A RAG system usually has two connected stages. First, a retriever searches a knowledge base for relevant chunks. Next, an LLM uses those chunks to produce an answer. An application can fail at either stage: the retriever may miss the correct passage, or the model may receive useful context but misunderstand it, omit important details, or invent unsupported claims.
Evaluating only the final answer makes these failures difficult to diagnose. A practical RAG evaluation therefore separates at least four questions:
- Was the relevant information retrieved? This tests retrieval quality.
- Was the supplied context sufficient? This tests whether the retrieved passages contain the evidence needed to answer.
- Did the answer follow the context? This tests faithfulness and helps identify hallucinations.
- Did the answer satisfy the user’s request? This tests correctness, completeness, format, and usefulness.
Keep a small, representative evaluation set under version control rather than relying on occasional manual checks. Each test case should record the user question, expected answer or answer criteria, relevant source passages, document identifiers, and any required citation. This makes results comparable when you change chunking, embeddings, reranking, prompts, models, or application logic.
For background on the infrastructure behind retrieval, see Best Vector Databases for RAG and Embedding Models Explained: How to Choose the Right Option for Search and RAG. Evaluation should be designed alongside those choices, not added only after an application produces unreliable answers.
Template structure
Use the following structure for each RAG evaluation run. It is simple enough for a spreadsheet and detailed enough to support automated testing later.
1. Define the evaluation scope
State what the application is expected to answer and what it must refuse or qualify. Record the knowledge-base version, retrieval settings, model, system prompt, and application release. Without this information, a score cannot be tied to a specific configuration.
2. Build the test set
Include realistic questions from the intended workflow, not just straightforward questions copied from source documents. A balanced set can contain:
- Questions with a clear answer in one document.
- Questions requiring evidence from several passages.
- Ambiguous questions that should trigger clarification.
- Questions about information that is absent from the knowledge base.
- Questions containing misspellings, abbreviations, or natural workplace wording.
- Adversarial or misleading questions designed to expose unsupported answers.
For each item, record a reference answer, acceptable answer points, relevant document or chunk IDs, and whether citations are required. A reference answer does not need to be the only valid wording. Define the facts that must be present and the claims that would make an answer incorrect.
3. Measure retrieval quality
Compare the retrieved chunks with the annotated relevant chunks. Useful measures include precision at k, which asks how much of the retrieved set is relevant, and recall at k, which asks whether the relevant material appears within the first k results. Mean reciprocal rank can help when the position of the first useful result matters.
These measures are diagnostic rather than complete quality scores. A high recall result does not guarantee that the model will use the evidence, and a low precision result may consume context space with distracting passages. Record the value of k and the retrieval configuration so that comparisons remain meaningful.
4. Check answer quality
Evaluate answers against separate criteria rather than one broad impression. Check correctness, completeness, relevance, clarity, instruction following, and appropriate uncertainty. For factual answers, compare individual claims with the retrieved evidence. For summaries or transformations, assess whether important information was preserved and whether unsupported additions appeared.
5. Check faithfulness and citations
Break an answer into factual claims and ask whether each claim is supported by the supplied context. Mark claims as supported, contradicted, unverifiable, or irrelevant. Then validate citations separately: does each citation point to the claimed source, and does the cited passage actually support the statement? A citation that merely comes from a relevant document is not necessarily evidence for every claim in the answer.
6. Record failures and actions
For every failed case, record the failure category, probable cause, severity, and proposed change. Common categories include poor chunking, missing documents, weak query formulation, irrelevant retrieval, context overload, prompt ambiguity, unsupported inference, citation mismatch, and formatting failure. This turns evaluation into an engineering loop rather than a pass-or-fail ceremony.
How to customize
Start with the risks that matter most to the application. A customer-support assistant may prioritise factual accuracy, refusal behaviour, and citation traceability. An internal document search tool may place more weight on recall and useful source links. A document summarizer may need detailed checks for omission, ordering, and unsupported conclusions. The evaluation criteria should reflect the consequence of an error, not simply the features that are easiest to measure.
Set explicit pass conditions before running a comparison. For example, you might require every critical question to retrieve at least one approved source, prohibit unsupported claims in high-risk answers, and require citations whenever the interface presents factual guidance. Avoid reducing the entire system to one blended score. A single average can hide a serious failure in a small but important category.
Use both automated and human review. Automated checks are useful for retrieval overlap, citation presence, required fields, latency, and regressions across a stable test set. Human reviewers are better suited to judging nuanced correctness, whether an answer is misleading despite being technically supported, and whether the response is useful to its intended audience. Give reviewers a short rubric with examples of acceptable and unacceptable answers.
When changing prompts, test the prompt as a component. Keep the system instructions, retrieved context, user question, output schema, and model settings visible in the test record. A prompt change that improves one task can weaken refusal behaviour, citations, or answer completeness elsewhere. Teams can use a versioned workflow such as the one described in Prompt Version Control: How Teams Should Track Changes, Tests, and Rollbacks.
Also test the boundaries of the knowledge base. Include questions about outdated, conflicting, duplicated, and missing information. If documents have dates or ownership metadata, decide whether the application should prefer newer or authoritative sources. Make that rule explicit in the retrieval and answer criteria instead of expecting the model to infer it.
Examples
Example: internal policy assistant
Suppose an assistant answers questions about an organisation’s leave policy. A test case might ask, “How much notice is needed for planned leave?” The record should identify the policy document, the relevant passage, the required notice period, and the citation expected in the answer.
Retrieval evaluation checks whether the correct policy section appears in the top results. Answer evaluation checks that the response states the requirement without adding an exception that is absent from the policy. Faithfulness evaluation checks every factual statement against the passage. Citation evaluation checks that the link or document reference points to the policy section rather than a general HR page.
Add a missing-information case such as, “Can unused leave be transferred to the next year?” if the knowledge base does not specify this. The desired behaviour may be to say that the available documents do not establish an answer and direct the user to the appropriate process. This case is valuable because a fluent guess can look helpful while being unsupported.
Example: product documentation assistant
For a technical documentation assistant, include a multi-step question that requires two sources, a version-specific question, and a question containing an informal abbreviation. Annotate which sources are necessary and whether the answer must include code, prerequisites, or warnings.
Review not only whether the final instructions are correct, but also whether the retriever found documentation for the requested version. If the system retrieves newer documentation for an older product version, the failure belongs partly to metadata filtering or retrieval design, not only to the generation prompt.
For a practical LLM application pattern, compare this approach with the review steps used in How to Build a Customer Support AI Assistant Without Training a Custom Model. You can also borrow claim-level review ideas from workflows for summarisation, such as How to Build a Document Summarizer with an LLM API.
When to update
Revisit the evaluation whenever a change could affect retrieval, generation, or user expectations. This includes adding or removing documents, changing parsing or chunking, replacing an embedding model, altering metadata filters, changing top-k or reranking settings, updating the LLM, editing a system prompt, changing the output format, or introducing a new user workflow.
Update the test set when real users expose a new question type or failure mode. Add the original question, remove sensitive information where necessary, label the failure, and define the expected behaviour before using it as a regression test. Periodically review whether the set still represents the application’s current users and documents; a test set can become misleading if it contains only old, easy questions.
After each material change, run a focused evaluation first, then the full regression set. Compare retrieval metrics, answer criteria, faithfulness, citation validity, latency, and cost as separate results. Investigate trade-offs rather than accepting an improvement in one measure as proof that the whole application is better.
To put this framework into practice, create a versioned evaluation file with one row per question and fields for expected evidence, retrieved IDs, answer, citations, scores, failure category, and reviewer notes. Run it against the current release, select a small set of high-risk cases for human review, and save the results with the application configuration. Then make one controlled change at a time, rerun the same cases, and keep any newly discovered failure as part of the next regression set. This gives your RAG application a repeatable quality process instead of an informal impression of whether it works.
For related guidance on reducing unsupported outputs, see How to Reduce Hallucinations in LLM Apps. The goal is not to claim that a RAG system is permanently accurate; it is to make its strengths, limitations, and changes visible enough for developers and teams to improve it responsibly.
