LLM costs become easier to control when you treat them as an engineering measurement problem rather than a surprise on an invoice. This guide shows how to estimate token spend, compare model options, use caching and batching appropriately, route requests by difficulty, and run a repeatable monthly review without sacrificing response quality.
Overview
The cost of an LLM application is usually driven by four variables: input tokens, output tokens, request volume, and the price attached to each token category. Some providers also distinguish between cached, uncached, batch, or priority processing. Your application may incur additional costs for tools, retrieval, storage, hosting, observability, or image and audio processing, but the model calculation is a useful starting point.
A simple cost model is:
Estimated cost = input tokens × input rate + output tokens × output rate
Apply the provider's billing unit consistently. If a rate is quoted per million tokens, divide the token count by 1,000,000 before multiplying. Keep input and output calculations separate because output tokens often have a different rate and are easier to increase accidentally through verbose prompts or unrestricted responses.
Cost optimisation should not mean selecting the cheapest model for every request. A low-cost model that produces unusable answers can increase total cost through retries, human review, support work, or failed transactions. The practical objective is to minimise the cost of an acceptable result. That requires measuring quality, latency, error rates, and cost together.
For related decisions, use the AI Model Selection Guide and the AI Tool Evaluation Framework.
How to estimate
1. Measure representative requests
Do not base a forecast on one short prompt. Select a sample that reflects real traffic, including short and long inputs, successful and failed calls, normal and edge-case conversations, and requests that use retrieval or tools. Record the following for each request:
- Input token count, including system instructions, user content, conversation history, retrieved context, and tool results.
- Output token count.
- Model and API mode used.
- Whether the request was retried, cached, batched, or routed to another model.
- Quality outcome, such as pass, fail, human correction, or escalation.
Calculate the average and the upper range separately. A mean can hide a small number of very long conversations that dominate spend.
2. Use a token-estimation worksheet
Create a spreadsheet or database with one row per request class. A useful worksheet has these columns:
| Request class | Requests per month | Avg input tokens | Avg output tokens | Input rate | Output rate | Estimated monthly cost |
|---|---|---|---|---|---|---|
| Support answer | Volume | Input estimate | Output estimate | Provider rate | Provider rate | Formula result |
| Document extraction | Volume | Input estimate | Output estimate | Provider rate | Provider rate | Formula result |
| Developer assistant | Volume | Input estimate | Output estimate | Provider rate | Provider rate | Formula result |
For each row, use:
Monthly cost = volume × ((average input tokens ÷ billing unit × input rate) + (average output tokens ÷ billing unit × output rate))
Then add a contingency line for retries, traffic growth, and unusually long requests. Label this as an assumption rather than presenting it as a guaranteed forecast.
3. Compare effective cost, not list price alone
A model comparison table should include more than advertised rates:
| Factor | What to record | Why it matters |
|---|---|---|
| Input pricing | Rate and billing unit | Long prompts and retrieved context can dominate spend. |
| Output pricing | Rate and billing unit | Verbose answers and agent loops can raise costs quickly. |
| Context limit | Published limit or documented range | Longer context may avoid summarisation, but can increase token usage. |
| Latency | Observed response time by request class | Slow responses may require queues, fallbacks, or user-facing trade-offs. |
| Quality | Pass rate on your evaluation set | A cheaper model is not useful if it fails important tasks. |
| Caching and batch support | Available modes and applicable pricing | Repeated prompts and non-urgent workloads may qualify for lower cost. |
Vendor pricing, context limits, caching features, batch discounts, and billing terminology change. Treat this table as a versioned internal record and verify every rate against the provider documentation before using it for a budget decision.
Inputs and assumptions
Your estimate is only as useful as its inputs. Separate measured values from assumptions so that a future review can replace guesses with production data.
Token inputs
- System prompt: Include it in every request if it is sent repeatedly.
- Conversation history: Count the retained history, not just the newest user message.
- Retrieved context: Measure the actual passages inserted by your RAG pipeline.
- Tool results: Include structured data, error messages, and repeated results returned to the model.
- Output limit: Record both the configured maximum and the typical output length.
Token counts vary by model and tokenizer, so character-based estimates are useful only for early planning. For a production forecast, capture token usage from API responses or the provider's usage records whenever possible.
Operational assumptions
Document monthly request volume, expected growth, retry rate, cache-hit rate, batch eligibility, and the proportion of requests assigned to each model. Also note whether a user interaction can trigger multiple calls. An agent workflow may make a planning call, a tool-selection call, a tool-result interpretation call, and a final response call. Counting only the visible response underestimates spend.
Caching and batching checklist
- Identify identical or near-identical prompts that occur repeatedly.
- Move stable instructions and reusable context into the part of the request that the provider can cache, where supported.
- Measure cache hits and misses rather than assuming every repeated request is discounted.
- Cache safe, non-personal results with an explicit expiry policy.
- Use batching for offline jobs such as classification, enrichment, evaluation, or document processing when latency is not important.
- Check whether batch results have different limits, availability, retention, or error-handling behaviour.
- Do not cache responses containing user-specific or sensitive information without an appropriate access-control design.
Prompt trimming is also valuable. Remove duplicate instructions, stale conversation turns, unnecessary retrieved passages, and verbose tool outputs. Keep the instructions that protect correctness, format, safety, and business rules.
Worked examples
Example: a two-model support workflow
Assume a support application receives 20,000 requests per month. A measured sample shows an average of 1,200 input tokens and 350 output tokens per request. The team tests a lower-cost model and a more capable model using the same evaluation set. The lower-cost model passes routine questions but fails a larger share of policy-sensitive cases; the capable model performs better on those cases.
Instead of assigning every request to the capable model, define two classes:
- Routine class: FAQ lookup, simple formatting, and clearly grounded answers. Route these to the lower-cost model.
- Complex class: ambiguous requests, multi-document reasoning, escalation summaries, and low-confidence retrieval. Route these to the capable model.
If 80% of requests are routine and 20% are complex, calculate the cost of each class independently using its measured token averages and current provider rates. Then compare that blended estimate with the all-capable-model baseline. Include the cost of routing checks, retries, and any fallback calls. The result is more useful than comparing two headline prices because it reflects the actual workflow.
Model-routing decision tree
- Is the request eligible for a cached answer? If yes, serve it after checking freshness and permissions.
- Is the task deterministic and better handled by code, a database query, or a conventional parser? If yes, avoid an LLM call.
- Is the task short, well-defined, and covered by evaluation tests? If yes, try the economical model.
- Does the request require long-context reasoning, nuanced judgement, or complex tool use? If yes, use the stronger model.
- Did the first model produce low confidence, invalid structure, or a failed evaluation check? If yes, retry selectively with a fallback model rather than repeating every request.
For a practical implementation, combine routing with the Prompt Testing and Regression Framework so that cost changes do not silently reduce answer quality.
When to recalculate
Run a lightweight cost review every month and a deeper review at least quarterly, or whenever a provider changes pricing, context limits, caching options, batch terms, model availability, or billing definitions. Recalculate sooner after a major prompt change, a traffic spike, a new agent loop, a retrieval configuration change, or a model migration.
Monthly cost-review workflow
- Export usage by model, request type, token category, and application environment.
- Compare actual input and output tokens with the previous estimate.
- Investigate anomalies such as sudden output growth, retry bursts, cache-hit declines, or unexpected model selection.
- Run the evaluation set against the current model and one plausible lower-cost alternative.
- Review representative requests for duplicated context, unnecessary history, and oversized tool results.
- Update the routing rules, token worksheet, and budget forecast.
- Record the date, pricing source, assumptions, quality results, and decision owner.
Keep a small change log. When spend rises, it should show whether the cause was higher traffic, longer prompts, a different model mix, a pricing change, or a workflow regression. For model switches, follow the LLM API Migration Guide and retest prompts before changing production traffic.
The most reliable LLM cost optimisation process is continuous: measure real usage, test quality, remove avoidable tokens, cache what is safe to reuse, batch what does not need immediate results, and route each request to the least expensive model that consistently meets its acceptance criteria.