"How to Safely Swap a Production LLM for a Cheaper Model"

# How to Safely Swap a Production LLM for a Cheaper Model

Last week an AI startup called Lindy told CNBC they ditched Claude entirely for

DeepSeek and saved millions. Their CEO said AI costs had become "unsustainable,"

exceeding personnel costs, and the switch was "a matter of survival for the

business."

A lot of teams read that story and asked themselves the same question: should I

do this too?

The answer is maybe. The problem is most teams skip the step that would tell

them for sure. They pick a cheaper model, run a few prompt-level checks, ship

it, and wait for the support tickets to roll in. Or they decide the risk is too

high and do nothing, burning money they do not need to burn.

There is a middle way. It looks like shadow traffic, an LLM judge, and a

defined quality threshold. Here is how to set it up.

The problem with gut-feel model swaps

If you swap one model for another and only test on five representative prompts,

you are essentially flying blind. The prompts you test are the ones you thought

of. The failure cases are the ones you did not.

Production LLM traffic follows a long tail distribution. A handful of request

types account for most of your volume, but the edge cases are where quality

breaks. A cheaper model like Haiku or DeepSeek V4 Flash might handle the top

80% of requests perfectly and then produce confidently wrong answers on the

remaining 20%. Without measuring that, you cannot say whether your swap

succeeded or failed.

The Lindy story makes this look binary: switch everything or switch nothing.

The better approach is gradual. Route a percentage of real traffic to the new

model, compare every response against the old one, and only promote when the

data says it is safe.

Shadow traffic: the low-risk swap

Shadow traffic means sending every request to both models simultaneously but

only using the old model's response for the user. The new model's response is

logged and compared. Users never see it. If it is bad, nothing breaks.

The implementation for most teams is simple. Intercept your LLM call at the

gateway or SDK layer. Fork the request payload to a second model call. Store

both responses alongside the request metadata (model name, prompt tokens,

latency, response time). Run your comparison offline or nearline.

You can start at 100% shadow coverage with zero user-facing risk. The cost is

double the inference bill during the evaluation period, but that is a temporary

investment that tells you exactly what a permanent swap would look like.

The LLM judge: who decides what "good enough" means

Comparing two model responses is not string matching. The same prompt can

produce semantically identical answers with completely different wording. You

need an evaluator that understands meaning, not just tokens.

The standard approach is an LLM-as-judge: a third model (typically Sonnet or

Opus) that receives the prompt, the old response, the new response, and a rubric

asking which is better, or whether the new response meets a quality bar.

A simple rubric might ask the judge to classify each pair as:

• Acceptable — the new response is functionally equivalent or better.
• Degraded — the new response is worse but not harmful.
• Critical — the new response is wrong, hallucinated, or harmful.

The judge does not need to be expensive. For most structured outputs

(classification, extraction, summarisation), a model at the Sonnet level is

enough to evaluate a cheaper model's output. The judge cost is a fraction of

your shadow traffic spend.

Run this evaluation across thousands of real requests, not handmade test cases.

The distribution matters. If you only see degradation on 3% of traffic but that

3% is your highest-value feature, you need to know before you ship.

Setting a quality threshold

Before you start shadow traffic, decide what "safe to ship" means for your

use case. There is no universal number. It depends on what your model does.

For classification tasks (routing, tagging, moderation), a 1% degradation in

accuracy might be acceptable if you save 60% on inference cost. For

user-facing chat in a medical or legal product, even 0.1% hallucination is too

high.

The pattern that works in practice is to run shadow traffic for a defined period

(one to two weeks of real traffic is usually enough to surface the long tail),

then compare the degradation rate against your threshold. If the new model

stays below the threshold, promote it. If it does not, you route around the

cases where it fails rather than abandoning the swap entirely.

What to do when the new model is not good enough everywhere

This is the insight that model cascading articles (including our earlier one on

the subject) cover in detail but is worth restating here: the answer to "is

this model good enough" is almost never a global yes or no. Every model has

tasks it handles well and tasks it struggles with.

When your shadow evaluation reveals specific failure patterns, you do not need

to abandon the swap. You build a router that sends the problematic cases to the

old model and everything else to the new one. Over time as models improve, the

failure categories shrink and you route more traffic to the cheaper tier.

The router itself can be a cheap model (Haiku, for example) running a

classification prompt. If the new model fails on multi-step reasoning but

succeeds on structured extraction, your router checks the request type and

assigns it accordingly. The cost of the routing call is negligible compared to

the savings from routing the easy cases to the cheap tier.

The Cost angle

This is where Cost fits into the workflow. When you run shadow traffic with an

LLM judge, you produce two kinds of data: cost data (how much each request

costs under each model) and quality data (whether the cheaper model passed the

judge eval). These two signals need to live in the same view for the trade-off

to be visible.

Cost gives you per-feature cost attribution. You can see which feature is

burning the most inference spend, route a percentage of its traffic to a

cheaper model, and compare the cost-quality ratio before and after. The

attribute-every-euro-to-a-feature model means you can decide to downgrade one

feature while leaving others on the premium tier without guessing whether the

savings are real.

The workflow looks like this:

1. Identify the most expensive feature or route in your Cost dashboard.
2. Configure a shadow route that sends a copy of its traffic to a cheaper

model.

1. Run an LLM judge against the paired responses.
2. Import the judge results to see the cost-quality trade-off per feature.
3. If the trade-off is acceptable, promote the cheaper model for that feature.
4. If not, build a sub-router that sends only the failing request types to the

premium model.

What the Lindy story actually tells us

The Lindy story is not a template to copy. It is a case study in the pressure

that builds when inference costs exceed every other line item. Crivello said

his company's AI costs exceeded personnel costs. That is a specific extreme

that forced an all-or-nothing decision.

Most teams are not in that position. They have time to evaluate, to measure,

and to route around failure cases. The ones that skip evaluation and swap

blindly are the ones that end up rolling back two weeks later when customers

notice quality drift.

When you decide to swap a production model, start with shadow traffic, an LLM

judge, and a defined quality threshold. That gives you an answer the data backs

up, which is a better position than "well Lindy did it so it must work."

Have you run a model swap in production? What did your evaluation process look

like?