Build a Retrieval-Augmented Generation App: A Practical RAG Tutorial

Overview

RAG is a useful architecture when an AI application needs access to information that is too large, too private, too specialised, or too changeable to place directly in a model prompt. Instead of asking the model to rely only on its trained knowledge, the application retrieves relevant passages from a controlled document collection and includes those passages in the request sent to the model.

A basic RAG pipeline has six stages:

1. Ingest: collect documents and extract readable text.
2. Prepare: clean the text, preserve useful metadata, and divide it into chunks.
3. Embed: convert each chunk into a numerical vector representing its meaning.
4. Index: store vectors and metadata in a vector database or another search system.
5. Retrieve: convert a user question into a query and select relevant chunks.
6. Generate: provide the retrieved context to an LLM and request an answer, ideally with citations.

The most important design decision is to treat retrieval and generation as separate components. A fluent answer does not prove that the right information was retrieved. Conversely, strong retrieval can still produce a poor answer if the prompt is ambiguous, the context is badly formatted, or the model is not instructed to handle missing evidence.

For a first version, choose one narrow use case, such as answering questions about an internal handbook, product documentation, or a defined set of technical notes. A constrained scope makes it easier to inspect failures and establish a useful evaluation set before adding more sources.

What to track

Document quality and freshness

Track each source with metadata such as title, owner, document type, version, publication date, last reviewed date, access scope, and source URL or identifier. This information supports filtering, citations, debugging, and later removal of obsolete content.

Record ingestion outcomes rather than silently skipping failures. Useful fields include the number of documents discovered, documents successfully parsed, pages or sections extracted, empty files, duplicate files, and files rejected because their format could not be processed. A document that was never indexed cannot be retrieved, so ingestion problems should be visible before users report missing answers.

Chunking and metadata

Chunking divides a document into retrieval units. Chunks that are too short may lose the surrounding explanation; chunks that are too long may contain several unrelated topics and consume too much context. There is no universal chunk size that works for every collection, so begin with a consistent strategy and test it against representative questions.

Preserve structure where possible. Headings, product names, section labels, timestamps, document versions, and access permissions often improve retrieval more than extra words do. Store these attributes alongside each vector. They can support filters such as “only search the current policy” or “only search documents available to this user”.

Retrieval behaviour

Measure more than the number of results returned. For a sample of questions, inspect whether the retrieved chunks contain the answer, whether they come from the correct source, and whether the top result is more useful than lower-ranked results. Track the query, filters, retrieved identifiers, similarity scores or ranking signals, and the final context sent to the model. Do not log sensitive content unnecessarily; apply access controls and retention rules appropriate to your application.

Consider testing several retrieval patterns:

• Semantic search: useful when the question and source use different wording.
• Keyword or lexical search: useful for exact names, codes, error messages, and identifiers.
• Hybrid search: combines semantic and lexical signals when both meaning and exact terms matter.
• Metadata filtering: limits results by source, date, product, customer, language, or permission.
• Reranking: applies a second relevance step to a smaller candidate set when the initial results need refinement.

Answer quality and citations

Track whether the answer is supported by the retrieved context, answers the actual question, follows the requested format, and acknowledges uncertainty when evidence is missing. A simple review label set can include supported, partly supported, unsupported, not answerable from the collection, and retrieval failure.

Ask the model to cite the source title, section, page, chunk identifier, or link that supports each material claim. Citations should be generated from metadata supplied by the application, not invented by the model. Also test citation correctness: a citation that points to a real document but does not support the statement is still a failure.

Cadence and checkpoints

A RAG system benefits from regular review because documents, embedding models, vector stores, prompts, and user questions can all change. Use a lightweight monthly check for an active application and a deeper quarterly review, adjusting the schedule to the risk and update rate of the source collection.

At ingestion time

• Confirm that new and changed files were detected.
• Check parsing errors, duplicate records, and unexpectedly empty text.
• Verify that access metadata and document versions were preserved.
• Remove or mark superseded chunks so old guidance is not retrieved by accident.

Monthly operational check

• Review failed requests, unanswered questions, and user corrections.
• Sample retrieval traces to see whether relevant chunks appear near the top.
• Compare answer latency, token usage, and error rates with the previous period.
• Inspect frequently repeated questions for missing documents or unclear terminology.
• Run a fixed evaluation set so changes can be compared consistently.

Quarterly design review

• Reassess chunking, metadata, filters, and the number of retrieved passages.
• Review whether the selected embedding, search, and model components still fit the use case.
• Test permissions with realistic user roles, including access to restricted sources.
• Review cost and latency by workflow rather than looking only at an overall average.
• Retire evaluation questions that no longer represent real usage and add new failure cases.

Keep changes versioned. Store the ingestion configuration, chunking rules, retrieval settings, system prompt, model identifiers, and evaluation results together. This makes it possible to compare releases and roll back a change that improves one category of questions while damaging another. The guidance in Prompt Version Control for Teams is useful when several people maintain the application.

How to interpret changes

When answer quality falls, resist the temptation to change the prompt first. Trace the request through the pipeline and identify the earliest stage that failed.

1. The document is missing: improve source coverage or ingestion scheduling.
2. The document is present but poorly extracted: fix parsing, OCR, tables, headings, or encoding.
3. The relevant text is split badly: adjust chunk boundaries and preserve section context.
4. The text exists but is not retrieved: test query rewriting, hybrid search, metadata filters, or reranking.
5. The right text is retrieved but ignored: simplify the context, remove duplicates, improve ordering, or revise the system prompt.
6. The answer is supported but poorly expressed: improve output instructions, citation formatting, or response validation.

Interpret metrics as signals rather than complete answers. A higher retrieval count may increase recall but add irrelevant context. A shorter response may reduce latency but omit necessary qualifications. A lower average cost may conceal poor performance on a high-value workflow. Review results by question type, source, user role, and failure category.

Keep a small set of “must not fail” questions. These should represent important tasks such as finding a required procedure, distinguishing two similar products, or identifying the current version of a rule. Run them after every meaningful change. For a broader evaluation process, see the RAG Evaluation Guide and Prompt Testing Frameworks guide.

Security and privacy checks belong in every review, not only in an initial launch checklist. Confirm that retrieval filters cannot be bypassed, that documents are not exposed through citations, and that logs do not retain sensitive text without a clear operational reason. If the application serves multiple teams or customers, test isolation with separate accounts and realistic permissions.

When to revisit

Revisit this RAG implementation whenever the underlying knowledge changes, the application’s users change, or the technical pipeline changes. At minimum, schedule a monthly operational review and a quarterly architecture review. Bring the review forward when a source collection receives a major update, users report unsupported answers, a model or embedding component is replaced, search performance changes, or access rules are modified.

Use this practical update checklist:

1. Export the latest ingestion, retrieval, latency, and answer-quality summaries.
2. Choose a representative sample of recent questions, including failed and corrected answers.
3. Trace each failure to ingestion, chunking, retrieval, prompting, generation, or presentation.
4. Make one controlled change at a time where possible, and record its expected effect.
5. Run the fixed evaluation set and compare results with the previous version.
6. Review permissions, citation accuracy, sensitive logging, and document freshness.
7. Publish the change notes, retain the evaluation results, and define the next review date.

Start small: one document collection, one retrieval strategy, one answer format, and a clearly labelled evaluation set. Once the system is dependable, expand its sources and capabilities gradually. A well-monitored RAG app is not finished when it produces its first convincing answer; it is maintained through observable ingestion, testable retrieval, disciplined prompt engineering, and regular review of the evidence behind every response.