RAG Tutorial: Build a Retrieval-Augmented Generation App Step by Step

Overview

Retrieval-augmented generation, or RAG, combines two separate capabilities. A retrieval system searches a collection of documents for relevant passages, while a language model uses those passages to produce an answer. Instead of relying only on information stored in a model's parameters, the application supplies selected context at query time.

A typical RAG pipeline looks like this:

1. Ingest: collect documents and extract usable text and metadata.
2. Clean: remove irrelevant formatting, duplicated content, navigation text, and extraction errors.
3. Chunk: divide documents into passages that can be searched and supplied to the model.
4. Embed: convert each chunk into a vector representation for semantic search.
5. Index: store vectors, text, and metadata in a vector database or another search system.
6. Retrieve: search the index when a user submits a question.
7. Generate: place the retrieved passages in a prompt and ask the language model to answer from that context.
8. Evaluate: measure retrieval quality, answer accuracy, latency, cost, and failure behaviour.

RAG is not a substitute for good application design. It cannot repair missing source material, ambiguous permissions, poor document extraction, or an evaluation process that only checks whether an answer sounds fluent. Treat the retrieval layer and generation layer as separate components so you can test and improve each one.

Checklist by scenario

Scenario 1: Building a small document question-answering app

Start with a narrow, well-defined collection such as internal guides, product documentation, or a small set of policy documents. Before choosing infrastructure, write down what a successful answer must contain and what the application should do when the answer is not present.

• Define the user question the app is intended to answer.
• Choose a limited document set with a clear owner and update process.
• Preserve useful metadata such as title, section, source URL, date, product area, and access group.
• Extract text while retaining headings, lists, tables where possible, and document boundaries.
• Begin with moderate, semantically coherent chunks rather than splitting at arbitrary character positions.
• Store the original text alongside each vector so retrieved results can be inspected.
• Retrieve several candidate passages, then test whether the relevant passage appears consistently.
• Use a prompt that instructs the model to answer only from supplied context and to say when the context is insufficient.
• Display citations or source labels when users need to verify the response.

A minimal generation prompt might look like this:

System: Answer using only the supplied context. If the context does not support an answer, say so. Do not invent policies, steps, or citations.

Context:
{retrieved_passages}

Question:
{user_question}

Answer:

Scenario 2: Building a production knowledge assistant

For a team-facing or customer-facing application, expand the checklist beyond semantic search. Access control, freshness, observability, and predictable failure handling become part of the RAG design.

• Apply document permissions during indexing and retrieval, not only in the user interface.
• Track document versions and remove or replace stale chunks when source files change.
• Use metadata filters for tenant, department, product, language, document type, or publication status.
• Consider combining keyword search with vector search when exact names, identifiers, error codes, or legal wording matter.
• Log the query, retrieved document identifiers, scores or rankings, model response, latency, and failure category without exposing sensitive content unnecessarily.
• Define limits for prompt length, retrieved passages, retries, and response time.
• Return a useful fallback when retrieval fails, the index is unavailable, or no result meets the relevance threshold.
• Maintain a test set of real and representative questions, including questions with no answer in the corpus.

Infrastructure choices vary. A managed vector database may reduce operational work, while a relational database with vector support may simplify an existing stack. A search engine can be useful where keyword, filtering, and ranking controls are already important. Compare options against your actual data size, security requirements, update frequency, team skills, and monitoring needs rather than selecting a database by reputation alone. For a broader comparison framework, see the guide to vector databases for RAG.

Scenario 3: Adding RAG to an existing LLM workflow

If you already have an AI summarizer, support assistant, meeting-notes workflow, or document-processing tool, add retrieval as a distinct step. First establish a baseline without retrieval. Then compare it with retrieval enabled using the same questions and output requirements.

• Identify which outputs require private, current, or organisation-specific information.
• Retrieve only the context needed for that task instead of attaching an entire document library.
• Keep extraction, retrieval, prompt construction, and generation separately configurable.
• Test whether retrieval improves factual support rather than merely increasing answer length.
• Use structured output when downstream software needs fields, labels, or actions.
• Require human review for decisions involving sensitive records, permissions, or consequential actions.

Workflows such as meeting notes often benefit from retrieval of project terminology, previous decisions, or approved templates, but retrieved context should not silently override the meeting transcript. Label the source of each piece of context so the model and the reviewer can distinguish current input from background knowledge.

What to double-check

Document preparation

Inspect extracted text before embedding it. Broken tables, repeated headers, missing section titles, scanned pages, and duplicated paragraphs can reduce retrieval quality. Keep a link from every indexed chunk back to its source document and location. This makes debugging and user verification much easier.

Chunking and metadata

Chunk boundaries should preserve meaning. A heading with the paragraphs that explain it is usually more useful than a fragment cut through the middle of a procedure. Test different chunk sizes and overlap rather than assuming one setting works for every document type. Store metadata that supports filtering and source display, but avoid adding fields that cannot be maintained reliably.

Embedding and search behaviour

Embedding models are not interchangeable in every application. Consider the language, terminology, document type, query style, hosting constraints, and expected update process. Test representative queries, including exact lookups, paraphrased questions, short queries, misspellings, and questions that span multiple sections. The embedding models guide provides a useful framework for making this decision.

Evaluation

Separate retrieval evaluation from answer evaluation. Ask whether the required passage was retrieved, whether irrelevant passages crowded it out, whether the answer is supported by the context, and whether the response follows the requested format. Include negative questions where the correct result is an explicit limitation. The RAG evaluation guide covers a fuller testing approach, including hallucination checks.

Prompt and cost controls

More context is not automatically better. Extra passages can distract the model, increase latency, and make responses harder to review. Set a retrieval limit, remove obvious duplicates, and define a consistent context format. Record usage and operational costs as part of testing, especially when the workflow may serve many users. The guide to evaluating AI tool pricing can help structure that review.

Common mistakes

• Indexing everything without classification: irrelevant, obsolete, or duplicate material can dominate search results. Begin with a controlled corpus.
• Treating vector search as a complete search strategy: exact identifiers and terminology may require keyword or hybrid search.
• Ignoring permissions: retrieval must respect the same access boundaries as the source system.
• Using a generic prompt: specify how to handle missing evidence, conflicting passages, citations, and uncertainty.
• Evaluating only fluent answers: a polished response can still be unsupported. Check its claims against retrieved evidence.
• Failing to refresh the index: document updates are incomplete if old chunks remain searchable.
• Hiding retrieval results: developers need retrieved text and metadata to diagnose failures.
• Changing several variables at once: alter chunking, search, prompts, or models in controlled experiments so improvements can be attributed.

When answers are unreliable, do not immediately replace the language model. First inspect the source text, retrieved passages, metadata filters, prompt assembly, and evaluation examples. Many apparent generation problems originate earlier in the pipeline.

When to revisit

Revisit your RAG application whenever its inputs or operating conditions change. At minimum, schedule a review before a major planning cycle, product release, content migration, or workflow change. Also review it after changing the embedding model, language model, vector store, chunking rules, access-control logic, or prompt template.

Use this compact maintenance checklist:

1. Confirm that indexed documents match the current source of truth.
2. Check for stale, duplicated, inaccessible, or incorrectly classified chunks.
3. Run the evaluation set and compare retrieval, answer support, latency, and cost with the previous version.
4. Review unanswered questions and add representative cases to the test set.
5. Inspect a sample of retrieved passages, not just final answers.
6. Verify that permissions and deletion workflows still work as intended.
7. Version prompts, retrieval settings, schemas, and model choices so changes can be rolled back.
8. Document the next change to test and the metric that will determine whether it helped.

For team projects, keep prompt and evaluation changes under version control; the prompt version control guide explains a practical approach. If the application is a support assistant, combine RAG checks with the design principles in this customer-support assistant guide. The goal is not to build the most elaborate pipeline. It is to maintain a dependable path from an authorised source document to an answer that can be checked.