Model Serving Architectures

The question is staleness, not modernity

Serving architecture debates tend to be about which pattern is most current. The useful question is narrower and answerable: how stale may this prediction be before it stops being useful?

Batch. Score everything on a schedule; write results to a table; read them at request time. Acceptable staleness: hours to days. Operationally the simplest thing that exists — a scheduled job, a table, and a lookup. No serving infrastructure, no latency budget, no autoscaling, and a failed run means yesterday's scores are still there.

Online. Score one request synchronously while someone waits. Acceptable staleness: none, because the input arrives with the request. This is the mode with real operational cost — latency budgets, autoscaling, warm capacity, timeout handling, graceful degradation.

Streaming. Score events continuously as they arrive on a bus. Acceptable staleness: seconds. Sits between the two, and inherits the operational characteristics of stream processing — consumer lag, replay semantics, exactly-once concerns.

Work through it with a churn model. Does a churn score need to reflect activity from the last five minutes? Almost never; the intervention is an email campaign that runs weekly. Batch. Cost drops by an order of magnitude and an entire class of on-call pages disappears.

Now a fraud model at checkout. The transaction being scored does not exist until the request arrives. Online, necessarily.

Then the interesting middle case, and the one worth remembering: a recommendation model. It feels like it must be online, because recommendations should feel fresh. But most of the work — user embeddings, candidate generation — depends on behavior that changes slowly. Precompute a few hundred candidates per user nightly in batch, then apply a cheap online reranker to the current session context. You get freshness where it matters at a fraction of the cost, and the expensive model runs where nobody is waiting for it.

Look for that hybrid before accepting an online architecture. The question "which part of this actually needs to be fresh" often turns an expensive serving problem into a table lookup plus a small model.

Budget the whole request

Teams optimize what they can see, and what they can see is the model. Then they discover the model was never the problem.

A typical online prediction request, on a 100 ms budget:

• Network and authentication overhead
• Feature lookup — often the largest single component, especially when several features come from different stores
• Model inference
• Postprocessing, response serialization, and logging

Feature lookup dominates more often than model inference does, and it is the piece nobody profiles. Six features from four different stores, fetched sequentially, each with a network round trip, is an easy 60 ms before the model has done anything. The fixes are ordinary engineering: batch the lookups, run independent fetches concurrently, colocate the feature store with the serving pods, cache features that change slowly.

Three practices make latency work tractable:

Set the budget from the product, then divide it. "Checkout must not feel slow" becomes "the fraud check gets 80 ms at p99," which becomes explicit allocations per stage. Without a number, every stage expands.

Measure p99, not the mean. Mean latency hides the tail, and the tail is what users experience as broken. A model with a 20 ms mean and a 400 ms p99 will generate complaints that the dashboard cannot explain.

Decide the timeout behavior before you need it. Every online model needs an answer to "what happens when this takes too long." Return a cached score? A rule-based default? Fail open or fail closed? For a fraud model, failing open means letting fraud through and failing closed means blocking legitimate customers — a product decision that belongs to a product owner, made in advance, not to an engineer at 2am.

Three levers, three different bills

When a serving system cannot keep up, there are exactly three responses, and they solve different problems.

Scale out — more replicas. Increases throughput linearly with cost. Does nothing for single-request latency: ten replicas serve ten times the traffic at exactly the same per-request speed. The right lever when you are throughput-bound. Watch out for cold starts — a model that takes ninety seconds to load makes reactive autoscaling useless, and you end up provisioning for peak anyway.

Scale up — bigger accelerator. Reduces single-request latency for large models. The right lever when you are latency-bound and the model is genuinely compute-heavy. The trap is utilization: a large accelerator sitting at fifteen percent during off-peak hours is expensive idle capacity, and accelerators are billed by the hour whether or not anything is running.

Shrink the work — quantize, distill, cache. The only lever that improves cost per request rather than buying more capacity. Quantization to lower precision often gives a substantial speedup for a small quality change. Distillation into a smaller model can be dramatic when the task is narrow. Caching removes the work entirely for repeated inputs.

The third lever has a cost the other two do not: it changes model behavior and therefore requires re-evaluation. A quantized model is a new model. It needs the same evaluation gates as any other candidate, including per-slice metrics — quantization damage is rarely uniform, and it tends to concentrate in exactly the rare cases you care about. Teams that skip this evaluation get their cost savings and a quality regression they discover from customer reports.

Before reaching for any of the three, check utilization. A serving deployment running at eight percent GPU utilization does not have a capacity problem; it has a batching or a routing problem, and adding replicas will make the bill worse without making anything faster.

Worked example: a recommendation service under budget pressure

A recommendation service costs too much and is too slow at peak. p99 is 340 ms against a 150 ms budget. The team is about to double the GPU pool.

Profile first. The breakdown at p99: 12 ms network and auth, 190 ms feature lookup, 95 ms inference, 43 ms postprocessing and logging. Doubling the GPU pool would address 95 ms of a 340 ms problem, and only the queueing part of it.

Feature lookup, 190 ms. Eleven features from three stores, fetched sequentially in a loop. Two changes: batch each store into a single multi-get, and issue the three store calls concurrently. Lookup drops to roughly 45 ms, bounded by the slowest store. No new infrastructure, no model change.

Postprocessing, 43 ms. Synchronous logging of the full feature vector and response to an analytics endpoint, in the request path. Moved to a fire-and-forget queue. Drops to 6 ms. The data is still captured — it is just no longer on the user's critical path.

Inference, 95 ms. Now the largest remaining piece, so it is finally worth attention. Utilization is 22 percent, which says batching, not capacity. Enabling dynamic batching with a 10 ms window raises utilization to 61 percent and moves inference to 70 ms — slightly slower per request, substantially better throughput.

Result. p99 is around 133 ms, inside budget, on the original hardware. The GPU pool was not doubled; it was later reduced by a third because the utilization improvement left headroom.

The general lesson is the ordering. Three of the four wins were plain engineering in code the ML team did not think of as ML work. The model was the last thing touched and the smallest share of the problem — which is the usual shape, and the reason profiling before scaling pays for itself every time.

Common mistake

The most common mistake is defaulting to online serving because it sounds like the real version. Online serving is the most expensive and most operationally demanding mode, and a large fraction of models scored in real time are consumed by workflows that would be perfectly happy with yesterday's numbers. Ask what breaks if the prediction is an hour old. If the answer is "nothing," you just deleted a service.

The second mistake is optimizing inference before profiling the request. Inference is the visible, interesting part, and it is frequently a minority of the latency. Measure the whole path first — the answer is usually feature lookup, and the fix is usually concurrency rather than hardware.

The correcting habit: for any serving system that feels too slow or too expensive, produce a stage-by-stage p99 breakdown and a utilization number before proposing any change. Those two figures decide which of the three levers applies, and they routinely reveal that none of them is needed.