LLM API Migration Guide: A Checklist for Switching Models Without Breaking Your App

Overview

An LLM API migration can affect nearly every layer of an AI application: authentication, request formats, system prompts, token accounting, structured output, tool calling, streaming, error handling, safety controls, observability, and user experience. Even when two providers expose similar chat-style APIs, their models may interpret instructions differently or support different parameters.

The safest approach is to treat migration as a controlled experiment. Keep the current integration available, create a provider-neutral interface inside your application, and compare the candidate model against representative production tasks before changing the default route.

What to preserve during a migration

• Output quality: accuracy, completeness, tone, formatting, and adherence to business rules.
• Reliability: valid JSON, predictable tool calls, appropriate refusals, and useful error behaviour.
• Performance: time to first token, total response time, timeout rate, and throughput.
• Economics: input and output token usage, retry volume, caching effects, and any operational overhead.
• Operational behaviour: logging, tracing, rate-limit handling, privacy controls, and rollback readiness.

For model selection criteria beyond migration mechanics, see the AI model selection guide. If your application depends on external knowledge, also test the interaction between the new model and your retrieval pipeline using this RAG tutorial.

How to estimate

Estimate migration impact using the same workload on the current and candidate systems. Avoid relying on a single prompt or an average response. Build a small evaluation set that reflects real traffic, including short and long inputs, difficult questions, malformed requests, tool-use tasks, and cases where the correct behaviour is to ask for clarification or decline.

A practical estimation formula

For a rough monthly cost comparison, use:

Estimated monthly cost = requests × [(average input tokens × input rate) + (average output tokens × output rate)] + retry cost + fixed operational costs

Use the rates and billing units published for the specific provider and model you are evaluating. Do not assume that similarly named models use the same context limits, tokenisation, caching rules, or billing treatment. If the provider charges in units other than tokens, adapt the formula accordingly.

For latency, record at least:

• Time to first token: how quickly streaming output begins.
• Total response time: the time until the response or tool workflow finishes.
• Tail latency: slower cases such as the 95th or 99th percentile, if your monitoring system supports them.
• Retry-adjusted latency: the time users experience after transient failures and retries.

For quality, score each test case against explicit criteria rather than asking whether the response “looks good”. A simple scale can work: 0 for failure, 1 for partial success, and 2 for a response that meets the requirement. Track separate scores for factual correctness, schema validity, instruction following, safety, and usefulness.

Use a decision worksheet

Create one row per workload and record current and candidate results. Useful columns include request type, input size, expected output, tool use required, current quality score, candidate quality score, current latency, candidate latency, token usage, failure category, and reviewer notes. This makes a model upgrade a repeatable engineering decision rather than a subjective demonstration.

Inputs and assumptions

Before changing code, document the assumptions that can alter the result. These inputs should be visible to developers, reviewers, and anyone responsible for approving the migration.

Provider compatibility comparison

Verify every cell against the provider’s current documentation before release. Model names, parameter names, context limits, pricing, supported features, and deprecation schedules can change. Treat this table as a review checklist, not a permanent compatibility guarantee.

Prompt and API adaptation checklist

• Separate system instructions, user content, examples, and retrieved context in your internal representation.
• Remove provider-specific parameters from business logic by placing them behind an adapter.
• Review system prompts for assumptions about formatting, verbosity, tool order, or refusal language.
• Define the exact output schema and validate it in application code.
• Handle missing fields, extra fields, invalid JSON, truncated output, and unexpected tool arguments.
• Use stable tool names and descriptions, and validate tool arguments before execution.
• Preserve request IDs, model identifiers, prompt versions, latency, token usage, and error categories in logs.
• Do not expose secrets or sensitive user content in evaluation datasets or debug logs.

If prompt changes are substantial, use a versioned system prompt and evaluate it separately from the model change. The prompt testing framework guide provides a useful structure for this work.

Worked examples

Example 1: Migrating a summarisation endpoint

Suppose an application summarises support tickets. The current endpoint returns a title, a short summary, priority, and suggested next action. The migration test set should include routine tickets, highly technical tickets, multilingual content if relevant, empty submissions, long transcripts, and tickets containing instructions that should be treated as data rather than commands.

Define acceptance rules before testing: the response must validate against the schema; the summary must not invent an action not present in the ticket; priority must use an allowed value; and the result must stay within the product’s length limit. Run the same cases through both models, then inspect failures by category. A candidate that is cheaper but frequently produces invalid priority values may create more engineering work than it saves.

Example 2: Estimating an agent migration

For a tool-using agent, count the complete workflow rather than only the first model call. One user request might produce a planning call, two tool-selection calls, a tool result, and a final response. Estimate:

Workflow cost = sum of every model call’s input and output cost + tool execution cost + retry cost

Test cases should include a successful single-tool path, a multi-step path, a tool failure, an invalid argument, a permission denial, and a request that should not call a tool. Compare not only final-answer quality but also whether the candidate chooses the correct tool, passes valid arguments, stops when the task is complete, and avoids repeated calls.

For multi-step systems, keep the old model available behind a feature flag. Start with internal traffic or a small controlled percentage, compare outcomes, and expand only when quality, latency, and error rates remain within your agreed thresholds. Review the AI agent tutorial for additional guardrail patterns.

Regression-test template

Test ID: MIG-001
Use case: Support-ticket summary
Input class: Long technical ticket
Expected behaviour: Valid schema; no invented resolution
Assertions: schema_valid, priority_allowed, summary_grounded
Metrics: quality_score, input_tokens, output_tokens, latency_ms
Failure action: Hold rollout and review trace

Store expected behaviour and assertions with the test, not just the prompt. This keeps the evaluation useful when prompts, models, or provider adapters change.

When to recalculate

Recalculate the migration decision whenever an underlying input changes. At minimum, review it before a model upgrade, after a provider announces a deprecation or parameter change, when pricing or context limits change, when traffic volume shifts, or when your application adds a new tool, language, document type, or safety requirement.

Rollback and post-launch checklist

1. Keep the previous provider or model configuration deployable.
2. Use a feature flag or routing layer so traffic can return to the previous path without a full code revert.
3. Define rollback thresholds in advance, such as schema-failure rate, task-quality score, latency, timeout rate, or cost per completed workflow.
4. Monitor model ID, prompt version, request volume, token usage, latency percentiles, retries, refusals, tool errors, and validation failures.
5. Sample outputs for human review while respecting privacy and retention requirements.
6. Compare results by use case instead of relying only on a single overall average.
7. Record the decision, test set, provider documentation reviewed, assumptions, and follow-up date.

Once the migration is stable, schedule a periodic review rather than treating it as finished forever. Keep the evaluation set current by adding anonymised examples from newly observed failure modes. For cost and latency improvements that do not require changing providers, see the token-saving playbook. A disciplined LLM API migration process lets you change models with evidence, preserve user trust, and identify quickly when a new option is not yet ready for production.