The Model Deprecation Playbook: migrate before the switch flips

Every model you build on will be switched off. Not deprecated in principle, not deprecated eventually — switched off, on a date, whether or not you are ready. OpenAI, Anthropic and Google have all retired production model versions with notice periods measured in months. Anyone running a real workload on a hosted model will go through this two or three times a year, permanently.

Most teams handle it the same way: ignore the email, remember it three weeks before the cutoff, swap the model string, watch quality quietly degrade, and spend the next month chasing complaints they cannot reproduce. That is not a technical failure. It is the absence of a process.

Here is the process. It takes about a day of real work spread over the notice period, and it converts a recurring emergency into a scheduled maintenance task.

Step 0: Build the inventory before you need it

You cannot migrate what you cannot find. Model identifiers leak into places nobody remembers: a config file, a hardcoded fallback, an environment variable in a staging box, a colleague’s automation, a Zapier step, a spreadsheet macro.

Make one table and keep it current. Five columns, nothing more:

• Where — repo and file path, or the name of the no-code tool.
• Model string — the exact identifier, including version suffix.
• What it does — one sentence a non-engineer understands.
• Blast radius — who sees a bad output, and how fast.
• Owner — one named person, never a team.

Find them with a single search across everything you control. The pattern is always some variant of the vendor’s naming scheme:

grep -rEn "(gpt-|claude-|gemini-|llama-|mistral-)[a-z0-9.\-]+" . \
  --include="*.py" --include="*.js" --include="*.ts" \
  --include="*.json" --include="*.yaml" --include="*.yml" --include="*.env*"

Run it on every repo, then walk the no-code tools by hand. The hand-walk is where the surprises live. Expect the inventory to be 30–50% larger than your mental model of it.

Step 1: Freeze a golden set on the old model — today

This is the step that everyone skips and everyone regrets. Once the old model is gone, you have no baseline. You cannot prove the new one is worse, and you cannot prove it is fine. You are arguing from memory against users arguing from memory.

So capture the baseline while the old model still answers. Pull 50–100 real inputs per use case — actual production traffic, not invented examples — run them through the current model, and store input and output together, with a timestamp and the exact model string.

Weight the sample deliberately: roughly 60% ordinary cases, 25% known-hard cases, 15% cases that have previously failed or generated a complaint. A golden set made only of easy inputs will pass every migration and catch nothing.

Store it as plain JSONL in version control. Not a notebook, not someone’s laptop.

{"id":"inv-014","use_case":"invoice_extract","input":"<raw text>",
 "baseline_output":"<model output>","model":"vendor-model-v2",
 "captured":"2026-08-21","tier":"hard"}

Step 2: Write down what “same” means

“Does it still work?” is not a testable question. Before you run a single comparison, define pass criteria per use case, in writing, with numbers. Three kinds:

• Hard constraints — things that must be exactly right. Valid JSON. Correct field names. Numbers matching source. Measured as a percentage; the acceptable failure rate is usually zero.
• Graded quality — things a human scores 1–5. Tone, completeness, whether the summary buried the important part. Score at least 30 samples blind.
• Operational — latency at p95, cost per thousand calls, refusal rate.

Write the criteria as a sentence you would be willing to defend: “Invoice extraction ships if hard-constraint pass rate is at or above the baseline, blind quality scores within 0.3 of baseline mean, p95 latency under 4 seconds, and cost per thousand no more than 20% above current.” If you cannot write that sentence, you do not yet understand what the system does.

Step 3: Run the comparison — blind

Run the golden set through every candidate model. Then have a human review outputs with the model identity hidden and the order shuffled. This matters more than it sounds: reviewers who know which output came from the shiny new model rate it higher, consistently, on identical text.

Grade the hard constraints automatically. Grade the quality dimension blind, two reviewers where the decision is expensive.

Then read the failures, individually. Aggregate scores tell you whether to worry. Only the individual failures tell you what broke — and the failure mode is almost never uniform degradation. It is a specific thing: the new model became more verbose, or started refusing a category it used to handle, or stopped emitting a field when the input was empty, or got noticeably better at English and worse at everything else.

Step 4: Fix the prompt before you accept the loss

Most migration quality drops are recoverable, because most prompts were tuned by trial and error against the old model’s quirks. Newer models generally need less scaffolding, not more.

Work in this order and re-run the golden set after each change, one variable at a time:

1. Delete the workarounds. Every “do not include a preamble” and “respond only with JSON” line you added to fight the old model. Many are now unnecessary and some actively hurt.
2. Re-anchor the format. Replace prose instructions about output shape with one concrete example of the exact output you want.
3. Cut the examples. If you were passing five few-shot examples, try two. Long example blocks crowd out the actual input.
4. Adjust temperature last. It is the smallest lever and the one people reach for first.

Budget two to four hours per use case. If you are past that and still below baseline, the answer is a different candidate model, not a fifteenth prompt revision.

Step 5: Ship on a ramp, with a switch

Never cut over all at once, and never ship without a way back. The deployment shape that works:

• Make the model a config value, not a constant. If reverting requires a code deploy, you will not revert at 11pm — you will argue about whether it is really that bad.
• Shadow first. Send a slice of live traffic to the new model, log the output, serve the old one. Zero user risk, real production distribution.
• Ramp 5% → 25% → 100%, holding at least 24 hours at each step so a full daily traffic cycle passes through.
• Define the rollback trigger numerically before you start. “Revert if hard-constraint failures exceed 1% or complaints exceed three in a day.” A trigger you invent during the incident is a trigger you will talk yourself out of.

Step 6: Close the loop

Delete the old model string from the inventory. Promote the new outputs to be the baseline for next time — this is the compounding step, because the golden set you built under pressure becomes the asset that makes the next migration routine. Write four lines: what broke, what fixed it, how long it took, what you would do differently. Put a calendar reminder to re-check the vendor’s deprecation page quarterly.

Failure modes to watch for

• Testing on invented inputs. Synthetic test cases are cleaner than real traffic and miss exactly the messiness that breaks things. Use production data.
• Migrating everything simultaneously. Do the lowest-blast-radius use case first, learn the failure pattern, then apply it to the rest.
• Accepting an average. “95% as good” can mean a uniform 5% dip or total failure on one customer segment. Only reading individual failures distinguishes them.
• Treating cost savings as free. A cheaper model that needs a verification pass or a human check is not cheaper.
• Silent fallbacks. Code that quietly retries a deprecated model will start failing at the cutoff with no signal. Make fallbacks log loudly.
• Skipping the ramp because the evals passed. Evals cover the distribution you thought of. Production covers the one you have.

The worked example

A five-person operations team runs three AI workloads: invoice field extraction, support-ticket triage, and weekly report drafting. Their vendor announces a 90-day sunset.

Day 1. They grep the repos and find seven references, not the four they expected — two in a staging config, one in a scheduled job nobody had touched in a year. Inventory written, owners assigned.

Day 2. They pull 80 real invoices, 90 real tickets, 40 report inputs and freeze the current outputs into a JSONL file in the repo.

Day 4. Pass criteria written. Invoice extraction is the strict one — a wrong number reaches a customer. Report drafting is loose; a human edits it anyway.

Day 9. Blind comparison against two candidates. Triage and reports pass immediately. Invoice extraction drops from 99% to 94% on hard constraints. Reading the 6% shows a single pattern: the new model returns null where the old one returned an empty string.

Day 10. One line added to the prompt specifying empty-string behaviour. Back to 99%.

Days 12–20. Shadow, then 5%, 25%, 100%, one workload at a time, lowest blast radius first.

Day 21. Done, with 69 days to spare, and a golden set that makes the next one shorter.

Total effort: roughly one working day, spread across three weeks. The alternative — the version where they wait until day 75 — costs more than that in a single afternoon of debugging invoice complaints they cannot reproduce.

Start here

• Run the grep today. Not this quarter — today. It takes four minutes.
• Pick your highest-blast-radius use case and freeze 50 real input-output pairs this week.
• Move every model identifier out of code and into config.
• Put a quarterly reminder in the calendar to check your vendors’ deprecation pages.

Deprecation is not an incident. It is a scheduled event that most teams choose to experience as an incident. The difference is a hundred saved examples and a written definition of “good enough.”