Reducing LLM spend is not simply a matter of choosing the cheapest model. This practical playbook shows teams and creators how to measure token use and latency, compress prompts, select only useful context, apply caching and batching, set sensible output limits, and route each task to an appropriate model without weakening the result.
Overview
AI API cost and response time usually come from several interacting factors: input tokens, output tokens, model rates, request volume, concurrency, retrieval context, tool calls, and provider-specific limits. A long system prompt may be harmless in a small experiment but expensive when it is sent thousands of times. Similarly, a workflow that makes four sequential model calls may feel slow even if each individual call is reasonably fast.
The goal of prompt cost optimization is not to make every prompt as short as possible. The goal is to remove material that does not improve the answer, reduce unnecessary work, and preserve the information required for accuracy, safety, and consistent formatting. The same principle applies to LLM latency optimization: measure the complete workflow, then improve the slowest meaningful stage.
Begin with a baseline for a representative period. Record requests, input tokens, output tokens, average and high-percentile response time, error or retry rate, and a quality score. A simple quality score can be based on human review, a test set, or task-specific checks such as valid JSON, correct citations, or successful tool execution.
For related implementation guidance, see the prompt testing framework guide and the AI agent workflow tutorial. Optimisation is safer when prompt changes are tested against a fixed set of realistic inputs rather than judged from one impressive response.
How to estimate
Use a per-request model before changing the workflow. The basic cost estimate is:
Estimated request cost = (input tokens ÷ 1,000,000 × input price) + (output tokens ÷ 1,000,000 × output price)
Use the input and output prices published for the specific model and billing unit you are evaluating. Do not assume that input and output tokens have the same rate. If a provider uses a different unit, convert the formula to that unit. Add separate terms for cached input, image or audio processing, tool calls, storage, retrieval, or other billable features where relevant.
For a monthly estimate, multiply the per-request result by the expected number of requests:
Monthly cost = cost per request × requests per month
For workflows with several model calls, calculate each stage separately and add the results. This exposes expensive steps that a single blended average can hide. Also create a conservative scenario using higher-than-average input, output, and request volumes. A workflow that appears affordable at its average may become difficult to manage when users submit unusually long documents or retry failed calls.
Latency needs a slightly different calculation. Track:
- Time to first token: how long the user waits before generation begins.
- Generation time: how long the model takes to produce the response.
- Total response time: the time from request submission to a usable result.
- Workflow time: the combined duration of retrieval, model calls, tools, validation, and retries.
Measure more than an average. Averages can conceal slow outliers, so review a high percentile alongside the median when the application is used interactively. Compare latency and quality after every significant change.
Inputs and assumptions
Build a small token-budget worksheet for each important workflow. Include these fields:
- Workflow name and purpose
- Model and provider
- Requests per day or month
- Average input tokens
- Average output tokens
- Maximum output tokens
- Number of model calls per user task
- Retrieved context size and selection method
- Cache hit rate, if caching is used
- Median and high-percentile end-to-end latency
- Quality checks and failure rate
Separate fixed prompt content from variable content. Fixed content may include system instructions, examples, schema definitions, and policy text. Variable content may include a user message, retrieved passages, conversation history, or a document. This distinction helps identify opportunities for caching, summarisation, or selective context inclusion.
Do not treat the model context window as a target. A larger available window does not mean every available document belongs in the request. For retrieval-augmented applications, retrieve fewer, more relevant passages and remove duplicates before generation. A useful RAG implementation should make context selection measurable. If retrieval quality is uncertain, consult the RAG evaluation guide before cutting context aggressively.
Set output limits according to the task. A classification, extraction, or short rewrite rarely needs the same output allowance as a detailed report. Pair the limit with an explicit format, such as a compact JSON schema or a maximum number of bullets. Limits should prevent runaway output, not truncate valid answers.
Keep assumptions visible. Provider pricing, model limits, caching features, and rate-limit rules can change. Record the date and source of each pricing input in the worksheet, then refresh it before making a purchasing or architecture decision.
Worked examples
Example 1: Compressing a support prompt
Assume a support assistant sends a 2,400-token system prompt, 1,100 tokens of conversation history, and an average 350-token user message. It produces 450 output tokens. The team discovers that the system prompt contains repeated instructions, unused examples, and a full glossary that applies to only a small share of requests.
After testing, the team reduces the fixed instructions to 1,200 tokens and includes glossary entries only when a classifier identifies a relevant topic. If the typical request now sends 1,200 system tokens, 700 history tokens, and 350 user tokens, the input falls from 3,850 to 2,250 tokens before any additional caching. The saving per request is calculated using the provider's current input-token rate. The quality comparison should check resolution accuracy, escalation decisions, policy compliance, and formatting—not only response length.
Example 2: Reducing a multi-step workflow
A meeting-notes workflow uses one call to summarise a transcript, a second to extract actions, and a third to rewrite the actions for a project system. If the second and third calls receive the complete transcript, they repeatedly pay for the same context and increase total latency.
A more efficient design can have the first call produce a compact, structured intermediate result containing decisions, actions, owners, dates, and uncertainty flags. The later calls receive that result rather than the full transcript. The team should compare the old and new workflow on extraction accuracy, missing actions, total tokens, total calls, and end-to-end time. Guidance on meeting-note workflows can help with the review steps: AI meeting notes workflows.
Example 3: Choosing where to spend optimisation effort
Imagine a creator tool that receives short requests but occasionally processes long documents. Prompt compression may have little effect on normal requests, while output limits and document chunk selection may have a large effect on long ones. A sensible experiment segments usage by task size, measures the cost and latency of each segment, and applies different rules rather than forcing one global prompt.
A practical optimisation decision tree is:
- If quality is already below the required level, improve instructions, context selection, or evaluation before reducing tokens.
- If input tokens dominate, remove repetition, trim history, summarise stable context, or retrieve fewer passages.
- If output tokens dominate, define the required format, set a suitable maximum, and avoid asking for duplicate explanations.
- If repeated inputs dominate, test prompt or result caching where the provider and workflow support it.
- If sequential calls dominate latency, combine compatible steps, run independent steps in parallel, or move simple checks into application code.
- If quality is stable across models, test routing simpler tasks to a smaller or faster model while reserving a stronger model for difficult cases.
- If failures and retries dominate, fix validation, timeouts, and request handling before optimising token counts.
When to recalculate
Revisit the worksheet whenever pricing inputs, model availability, context limits, caching terms, rate limits, or benchmark results change. Also recalculate after a prompt rewrite, a new retrieval strategy, a change in traffic, or the addition of tools and agent steps.
For an established workflow, schedule a regular review based on its importance and volume. High-volume production tasks deserve more frequent monitoring than occasional experiments. Create a small dashboard or log report showing tokens per request, cost per successful task, latency, retries, and quality failures. “Cost per successful task” is often more useful than cost per request because a cheap response that needs correction or repetition is not genuinely efficient.
Before shipping an optimisation, run a fixed evaluation set containing short, typical, long, ambiguous, and adversarial inputs. Compare the baseline with the proposed version and record both improvements and regressions. Keep a rollback version of the prompt and configuration.
Finally, make optimisation part of normal prompt engineering rather than a one-off cost exercise. Start with one high-volume workflow, establish its token budget, remove waste in small steps, and review the quality evidence after each change. This approach helps teams reduce LLM costs and improve responsiveness while keeping the output useful, predictable, and fit for the real work it supports.
