Safe Deployment Strategies for Models

Three strategies, three different questions

Shadow, canary, and blue-green get discussed as alternatives. They are not. They answer different questions, and a mature deployment uses all three at different moments.

Shadow answers: does it work? Mirror live traffic to the candidate, compare its outputs against the current model, and throw its responses away. Zero user exposure. It finds crashes, memory problems, latency regressions, and prediction disagreement on real traffic — which is categorically better than your test set, because production traffic contains the malformed, the unusual, and the seasonal.

What shadow cannot tell you is whether the new model is better. You see that it disagrees on eight percent of requests. You cannot see who was right, because nobody acted on the shadow predictions and so no outcomes exist.

Canary answers: is it better? Route a small slice of real traffic and measure business outcomes. This is the only way to learn whether the model improves the thing you actually care about. It requires real user exposure, which requires a fast automatic rollback and a slice small enough that being wrong is survivable.

Blue-green answers: can we get back? Keep the previous version fully warm and switch traffic between them at once. Fast, complete rollback. Costs a full standby environment.

They compose in the obvious order. Shadow first to catch operational faults with zero risk. Then canary to measure value on a small slice. Blue-green underneath as the escape hatch. Each stage answers its own question and hands a narrower set of unknowns to the next.

Wiring shadow traffic correctly

Shadow deployment is conceptually simple and has two implementation failures that turn a zero-risk technique into an outage.

Failure one: the shadow call is not truly asynchronous. If the request handler awaits the shadow response, or if the shadow model shares a connection pool, thread pool, or GPU with the primary, then a slow or crashing candidate degrades the user path. The whole premise is violated. The shadow call must be fire-and-forget, resource-isolated, and bounded by a timeout that discards rather than retries.

async def handle(request: Request) -> Response:
    features = await build_features(request)
    response = await primary_model.predict(features)   # the user's answer

    # Never awaited. Failures here are logged and dropped, never raised.
    # Isolated client and pool so the candidate cannot starve the primary.
    asyncio.create_task(shadow_compare(features, response, request.id))

    return response

Failure two: the shadow path has side effects. If the candidate's tool calls, writes, or notifications reach real systems, shadow traffic doubles them. The classic version of this is a shadow-mode recommendation service writing impression events into the same analytics table, which corrupts the very metrics you are deploying to protect. Anything downstream of the shadow model must be stubbed or routed to a separate sink.

With the wiring correct, four comparisons are worth computing:

Disagreement rate. What fraction of requests produce a materially different output. A near-zero rate on a model you expected to change things means something is misconfigured — often the candidate silently falling back to the same weights.

Disagreement by slice. Where the models differ most. Concentrated disagreement in one segment is the most informative signal shadow mode produces, and it is invisible in the aggregate.

Latency distribution. Full percentiles under real traffic shape, not a load test.

Error rate. Including malformed inputs the candidate rejects and the primary tolerated, which is where schema changes surface.

Run shadow long enough to cover a full traffic cycle. A candidate that looks fine on Tuesday afternoon and falls over during Monday's batch window is a common and entirely avoidable discovery.

Canary: define the trigger before you start

Canary is where real users are exposed, so the discipline is about deciding in advance.

Choose the metric that matters, and accept the wait. Model quality metrics are available immediately; business metrics may take hours or days. Resist the temptation to promote on the fast proxy alone. If conversion is the goal, conversion is the metric, and the canary runs long enough to measure it.

Write the rollback trigger as a number, before deploying. "Roll back if p99 latency exceeds 250 ms for five minutes, or if error rate exceeds 0.5 percent, or if conversion in the canary slice drops more than two percent with statistical significance." A precise trigger can be automated. A vague one becomes a judgment call by whoever is awake, and that person will hesitate, because rolling back feels like admitting failure.

Size the slice for detectability. One percent sounds prudent and may take weeks to reach significance on a low-frequency metric. Compute the sample size you need first; the answer determines both the slice and the duration.

Assign traffic consistently. Route by a stable hash of user id, not per request. A user who sees the new model on one page and the old one on the next experiences inconsistency, and their behavior pollutes both arms of your measurement.

Rollback is a property you build in advance

"We can roll back" is asserted constantly and true less often than teams believe. Four preconditions, each of which fails in practice.

The previous artifact is still deployable. Pinned by digest, stored, and warm. Not "we can rebuild it from the branch," because the branch has moved and the build may no longer produce the same thing. This is where floating tags hurt: if model:prod moved, you need a record of what it pointed at before.

Schema changes are backward compatible. If the new model required a feature payload change and the serving layer was updated to match, the old model can no longer parse what it receives. Rolling back the model without rolling back the payload produces a different outage. Deploy schema changes as additive, in a separate release, ahead of the model that needs them.

A trigger metric exists and is monitored. Rollback capability without a signal is rollback that happens after a customer complains.

No irreversible side effects have occurred. This is the one that cannot be engineered away. If the new model spent four hours sending emails, approving refunds, or writing to a shared table, rollback restores the code and not the consequences. For models with irreversible effects, the canary slice must be small enough that the damage is absorbable, because rollback is not a remedy.

Test all four in a drill. Pick a low-traffic period, roll back a production model deliberately, and time it. Teams that run this drill routinely discover that step one takes forty minutes because the previous image was garbage-collected — which is much better learned on a Tuesday afternoon than during an incident.

Worked example: promoting a pricing model

A pricing model with real money attached. Walk the full sequence.

Week 1, shadow. The candidate runs against mirrored traffic for seven days, covering a full weekly cycle. Two findings. First, p99 latency is 180 ms against the current model's 90 ms — the candidate uses a larger feature set with an extra store lookup. Second, disagreement is 11 percent overall but 34 percent for enterprise accounts. The latency issue is fixed by batching the extra lookup. The enterprise disagreement is investigated and turns out to be legitimate: the candidate was trained on more recent enterprise data and the old model was stale. Nobody knew that until the slice comparison showed it.

Week 2, canary at 5 percent. Traffic assigned by stable user hash. Trigger defined in advance: roll back if quote-to-close drops more than 1.5 percent with significance, if p99 exceeds 200 ms for ten minutes, or if any manual price override rate doubles. That last one was added because a bad pricing model shows up first as salespeople overriding it, which is a faster signal than close rate.

Day three, override rate in the canary rises by 60 percent — not enough to trigger, enough to investigate. Overrides concentrate in one region where the candidate quotes systematically low. Root cause is a currency conversion applied at a different point in the pipeline. Fixed, and the canary restarts.

Week 3, canary re-run. Metrics hold. Close rate up 2.1 percent with significance, overrides flat, latency inside budget.

Promotion. Blue-green switch with the previous version kept warm for seven days. Automatic rollback triggers stay armed for the full week, because the canary slice never included the largest customers and their behavior is the remaining unknown.

Total elapsed time: three weeks for a model change that took two days to train. That ratio bothers people, and it should not. The override-rate finding in week two would have been a revenue incident affecting every customer instead of five percent of them, and it would have been found by a salesperson rather than a dashboard.

Common mistake

The most common mistake is treating shadow mode as sufficient evidence to promote. Shadow proves the candidate runs correctly and shows where it disagrees. It cannot show that the disagreements are improvements, because no outcome was ever observed for the shadow predictions. Promoting on shadow results alone is promoting on the assumption that different means better.

The second mistake is deploying without a written rollback trigger. Without a number, the decision to roll back gets made by a person under pressure who is weighing "this might be normal variance" against the embarrassment of reverting. That person will wait, and waiting is exactly what the trigger exists to prevent.

The correcting habit: before any model deployment, write two sentences. What metric would tell me this is worse, and at what threshold do I roll back automatically? If either sentence is hard to write, the deployment is not ready, and the difficulty is telling you something useful about what you failed to instrument.