Choosing between prompting, retrieval-augmented generation (RAG), and fine-tuning is an architecture decision, not a contest to find the most advanced option. This guide gives you a repeatable way to scope an AI application, compare likely costs and trade-offs, and decide whether to start with prompt engineering, add retrieval, fine-tune a model, or combine these approaches.
Overview
These three approaches solve different problems:
- Prompting changes how you instruct a model at request time. It is usually the fastest and simplest place to start.
- RAG gives a model relevant information retrieved from your documents, database, or other approved sources. It is useful when answers must reflect changing or private knowledge.
- Fine-tuning adapts model behaviour using examples prepared for a particular task, format, tone, or domain pattern. It can improve consistency, but requires suitable training data and an evaluation process.
The key distinction is between knowledge and behaviour. If the model needs access to facts that change, RAG is often the more direct solution. If it needs to follow a recurring style or produce a reliable structure, prompting or fine-tuning may be more appropriate. If the problem is unclear instructions, weak examples, or poor evaluation, changing the architecture may not help until the prompt and test set improve.
| Approach | Best suited to | Data needed | Typical cost and latency considerations | Maintenance | Output control |
|---|---|---|---|---|---|
| Prompting | Instructions, classification rules, extraction, summarisation, structured responses | Task description, examples, constraints, and test cases | Usually the lowest implementation cost; prompt length affects request size and latency | Revise prompts and regression tests | Moderate; improved with schemas, examples, and validation |
| RAG | Private, current, or large collections of reference material | Documents or records, chunking strategy, embeddings or search index, retrieval tests | Additional retrieval work can increase latency and infrastructure complexity | Maintain the source data, index, permissions, and retrieval quality | Good factual grounding, but dependent on retrieved context |
| Fine-tuning | Repeatable behaviour, style, classification boundaries, or a specialised response format | Curated examples, a supported base model, training and evaluation process | Training adds setup cost; serving cost and latency depend on the resulting model and platform | Manage datasets, versions, retraining, and model evaluations | High behavioural consistency when the dataset represents the task well |
This comparison is directional rather than a price list. Model context limits, fine-tuning availability, hosted features, and pricing vary by provider and can change. Confirm the current capabilities of the model and platform you plan to use before committing to an implementation.
How to estimate
Use a staged decision process instead of selecting an approach from the start.
- Define the task and failure. Write down the input, desired output, acceptable error, and the consequence of an incorrect answer. “Build a support assistant” is too broad; “classify incoming tickets into six queues and return a confidence flag” is testable.
- Create a small evaluation set. Gather representative examples, including difficult cases and cases where the system must refuse or escalate. Record the expected output, evidence requirements, and any fields that must be exact.
- Build a prompting baseline. Use clear instructions, a defined output format, a few relevant examples, and explicit handling for missing information. Test the baseline before adding components.
- Check whether the missing capability is knowledge or behaviour. Missing or changing facts point towards RAG. Inconsistent formatting, tone, or task behaviour may be addressed with better prompting, validation, or fine-tuning.
- Add only the smallest useful component. Introduce retrieval if the baseline lacks source knowledge. Consider fine-tuning only after you have enough high-quality examples and evidence that prompting plus validation is insufficient.
- Compare total system performance. Measure answer quality, groundedness, refusal behaviour, latency, token usage, operational effort, and failure recovery. A more complex system is not automatically better.
A simple decision tree is:
- If the task is mainly instruction following, extraction, transformation, or summarisation, start with prompting.
- If the answer depends on private, current, or too-large-to-include information, test RAG.
- If the task requires stable behaviour across many examples and you have a representative dataset, investigate fine-tuning.
- If the task needs both current knowledge and consistent behaviour, use RAG plus prompting, then evaluate fine-tuning as an optional later step.
- If errors come from poor retrieval, ambiguous requirements, or weak evaluation, fix those issues before fine-tuning.
For a practical implementation sequence, begin with a prompt test suite, then add retrieval evaluation if needed. The prompt testing frameworks guide covers repeatable prompt evaluation, while the RAG evaluation guide focuses on retrieval quality, answer accuracy, and hallucination checks.
Inputs and assumptions
Before estimating effort or selecting an architecture, capture these inputs:
- Knowledge volatility: How often do the facts change, and must responses reflect updates immediately or on a scheduled cycle?
- Knowledge ownership: Are the relevant facts in documents, databases, APIs, or user-provided files? Can the system retrieve only content the user is authorised to see?
- Task repetition: Is the system performing one stable task or a broad set of tasks with different instructions?
- Output strictness: Does a human review the answer, or must the response pass a schema, workflow, or downstream parser?
- Data volume and quality: For RAG, assess document cleanliness, duplication, metadata, access rules, and searchability. For fine-tuning, assess example accuracy, coverage, consistency, and labelling.
- Latency target: Decide whether an extra retrieval step is acceptable and whether long prompts or large contexts create a measurable delay.
- Privacy and deployment requirements: Identify what data may be sent to a model provider, where indexes are hosted, and how logs are handled. Treat these as requirements to verify, not assumptions about any particular platform.
- Change management: Estimate who will update prompts, documents, indexes, datasets, evaluations, and model versions.
For a rough operating estimate, separate one-time work from recurring work:
Total effort = initial build + evaluation and integration + recurring data or model maintenance + monitoring and incident response.
This formula is deliberately broader than an API bill. Prompting may have low infrastructure effort but still require careful testing. RAG adds indexing, retrieval, permissions, and source maintenance. Fine-tuning adds dataset preparation, versioning, training runs, and re-evaluation when the base model or task changes.
Worked examples
Example 1: Extracting fields from invoices
Suppose an application receives invoices and must return supplier name, invoice date, total, and purchase-order number. Start with prompting and structured output. Add input validation and a review path for missing or ambiguous fields. RAG is unnecessary unless the task also requires looking up supplier records or applying policies stored outside the invoice. Fine-tuning may become relevant only after a substantial evaluation set shows a repeatable pattern of errors that clearer instructions, examples, and validation do not solve.
Example 2: An internal policy assistant
An assistant answering questions about internal policies needs access to controlled, changing documents. Prompting alone cannot reliably supply information that is not in the model’s context. A RAG design is the natural baseline: retrieve relevant policy passages, include source metadata, instruct the model to answer only from the supplied evidence, and provide an escalation path when evidence is missing or contradictory. Fine-tuning might later improve response style or classification, but it should not be treated as the primary mechanism for keeping policy facts current.
Example 3: Consistent support-ticket classification
A support team wants every ticket assigned to one of a fixed set of categories with a short reason. Prompting, examples, a strict schema, and a test suite may be enough. If the labels are stable and the system repeatedly confuses similar categories despite high-quality examples and clear definitions, fine-tuning could be evaluated. RAG is useful only if classification depends on a changing knowledge base, such as product entitlement or account-specific information.
Hybrid systems are common. A useful pattern is a prompt that defines the task, RAG that supplies current evidence, and deterministic code that validates fields and applies business rules. See the practical RAG tutorial for an implementation path, and the cost and latency optimisation guide for ways to reduce unnecessary context and request overhead.
When to recalculate
Revisit the decision whenever an input changes, not only when the application fails. Recalculate after a model or provider changes its context limits, fine-tuning support, supported modalities, or pricing structure. Recalculate when document volume, update frequency, permissions, traffic, or latency requirements change. A larger corpus can expose retrieval weaknesses; a new model can change the value of existing prompts or fine-tuned versions.
Use this short review checklist:
- Run the current evaluation set against the production baseline.
- Compare prompt-only, RAG, and fine-tuned candidates on the same examples where each option is technically suitable.
- Record quality, unsupported claims, refusal behaviour, latency, token usage, operational effort, and failure recovery.
- Check whether the data and permissions assumptions are still valid.
- Keep the simplest approach that meets the acceptance criteria, and document why more complex options were rejected.
The practical default is straightforward: start with strong prompt engineering and a measurable test set. Add RAG when the system needs controlled, changing knowledge. Consider fine-tuning when the behaviour remains inconsistent, the task is stable, and a representative dataset supports training. Revisit that choice whenever your model, data, traffic, or quality target changes.
