Serving Models at Scale

From Model to Production System

You've trained a model, compressed it, and measured 50 ms latency on a GPU. Now what? Deploying that model to production isn't just running it on a server. You need:

• Concurrency handling: Multiple requests arriving simultaneously
• Batching: Grouping requests to maximize hardware utilization
• Latency guarantees: 95th percentile latency must stay under your SLA
• Model versioning: Roll out new models without downtime
• Monitoring: Track inference latency, error rates, and data drift
• Fallback and graceful degradation: What happens if the model server crashes?

This lesson covers the engineering discipline of serving ML models at scale. You'll see concrete code examples, understand the architecture, and learn the operational practices that separate hobby projects from production systems.

Batching and Latency-Throughput Tradeoff

A naive approach: each client sends one request, the server runs inference immediately and returns the result. If a single inference takes 50 ms, you handle at most 20 requests/second. If you have 10 requests arriving simultaneously, 9 of them wait while the server processes them one-by-one.

Batching is the key optimization: accumulate requests for a short time (e.g., 10 ms), then run inference on all of them together. Modern GPUs are optimized for batch processing; batch size 32 might be 10x faster per-sample than batch size 1.

Trade-off: Batching reduces per-sample latency but increases latency for early requests in a batch. If you wait 50 ms for a batch, request #1 experiences 50 ms additional latency.

Here's a FastAPI-style example with batching:

import asyncio
import time
import numpy as np
from fastapi import FastAPI
from pydantic import BaseModel
import torch

app = FastAPI()
model = torch.load("model.pt")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device).eval()

# Batch queue and processing
request_queue = asyncio.Queue()
batch_size = 32
batch_timeout_ms = 20  # Wait up to 20 ms to fill batch

class InputData(BaseModel):
    features: list[float]

class PredictionResult(BaseModel):
    prediction: float
    latency_ms: float

async def batch_processor():
    """Continuously process batches from the queue."""
    while True:
        batch = []
        batch_times = []
        deadline = time.time() + batch_timeout_ms / 1000

        # Collect requests until batch is full or timeout
        while len(batch) < batch_size:
            wait_time = deadline - time.time()
            if wait_time <= 0:
                break

            try:
                future, request_time = await asyncio.wait_for(
                    request_queue.get(),
                    timeout=wait_time
                )
                batch.append(future)
                batch_times.append(request_time)
            except asyncio.TimeoutError:
                break

        if batch:
            # Run inference on batch
            batch_inputs = torch.tensor(
                [f.input.features for f in batch],
                dtype=torch.float32,
                device=device
            )

            with torch.no_grad():
                inference_start = time.time()
                batch_outputs = model(batch_inputs)
                inference_time = (time.time() - inference_start) * 1000

            # Return results
            for i, future in enumerate(batch):
                elapsed_time = (time.time() - batch_times[i]) * 1000
                future.request.output = PredictionResult(
                    prediction=batch_outputs[i].item(),
                    latency_ms=elapsed_time
                )
                future.set_result(None)

# Start batch processor in background
@app.on_event("startup")
async def startup():
    asyncio.create_task(batch_processor())

@app.post("/predict")
async def predict(data: InputData) -> PredictionResult:
    """Submit a request and wait for batched inference."""
    future = asyncio.get_event_loop().create_future()
    future.input = data
    await request_queue.put((future, time.time()))
    await future
    return future.request.output

Behavior:

• Request #1 arrives at t=0 ms, joins queue, waits 20 ms for batch to form
• Requests #2-30 arrive during the 20 ms window, join batch
• At t=20 ms, batch of 30 runs inference (50 ms)
• At t=70 ms, all 30 requests get responses
• Request #1 experiences total latency: 70 ms (20 ms batch wait + 50 ms inference)
• Without batching, request #1 would experience 50 ms (just inference)

So batching added 40 ms to request #1's latency, but throughput increased from 20 req/s to 1500 req/s (30 requests in 70 ms). This is the core tradeoff: lower latency for individual requests, or higher total throughput.

Set your batch size and timeout based on SLA. If your SLA is "p95 latency < 100 ms," you can afford a 20-50 ms batch wait.

Model Versioning and Safe Rollouts

You've trained model v2, which is 1% more accurate than v1. You want to deploy it to production. But what if v2 fails in unexpected ways on real data? You need a rollout strategy.

Canary deployment: Route a small percentage of traffic (5-10%) to the new model while keeping the rest on the old. Monitor metrics for the canary. If error rates or latency spike, roll back immediately. If metrics look good after a time window (hours), gradually increase traffic to the new model.

# Pseudo-code for canary deployment
import random

def route_request(request):
    """Route to model v1 or v2 based on canary traffic percentage."""
    canary_percentage = 5  # 5% to v2, 95% to v1

    if random.random() < canary_percentage / 100:
        model = model_v2
        request.metadata['model_version'] = 'v2'
    else:
        model = model_v1
        request.metadata['model_version'] = 'v1'

    prediction = model.predict(request.features)
    return prediction

# Monitor metrics separately per version
# In your logging/monitoring system:
# - latency_p95[v1], latency_p95[v2]
# - error_rate[v1], error_rate[v2]
# - prediction_distribution[v1], prediction_distribution[v2]

# If v2 metrics are worse, set canary_percentage = 0 (instant rollback)
# If v2 metrics are good for 1 hour, increase canary_percentage to 25%
# Gradually ramp up until v2 reaches 100% of traffic

Production serving systems (TorchServe, TensorFlow Serving, Seldon) have built-in canary, shadow, and A/B testing deployment patterns. Shadow deployment runs both models on the same request but only returns the old model's prediction, letting you compare outputs offline.

Monitoring and Observability

You must monitor several dimensions:

Inference latency:

• p50, p95, p99 latencies (percentiles matter more than averages)
• Alert if p95 exceeds your SLA

Error rates and failures:

• Inference errors (model crashes, out of memory)
• Request timeouts
• Model loading failures

Model performance:

• Prediction distribution (are outputs drifting?)
• User feedback (for some domains, you can log user-approved labels post-hoc)
• Subgroup performance (are some demographic groups getting worse predictions?)

System health:

• GPU/CPU utilization
• Memory usage
• Number of concurrent requests
• Batch processing time

Here's a monitoring example using FastAPI + Prometheus metrics:

from prometheus_client import Counter, Histogram, Gauge
from fastapi import FastAPI
import time

app = FastAPI()

# Define metrics
prediction_counter = Counter(
    'model_predictions_total',
    'Total predictions',
    ['model_version', 'prediction_class']
)

prediction_latency = Histogram(
    'model_prediction_latency_seconds',
    'Inference latency in seconds',
    ['model_version'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0]
)

inference_errors = Counter(
    'model_inference_errors_total',
    'Total inference errors',
    ['model_version', 'error_type']
)

active_requests = Gauge(
    'model_active_requests',
    'Number of active requests',
    ['model_version']
)

@app.post("/predict")
async def predict(data: InputData):
    model_version = request.headers.get("X-Model-Version", "v1")
    active_requests.labels(model_version=model_version).inc()

    try:
        start_time = time.time()
        prediction = model.predict(data.features)
        latency = time.time() - start_time

        # Log metrics
        prediction_latency.labels(model_version=model_version).observe(latency)
        prediction_counter.labels(
            model_version=model_version,
            prediction_class=str(int(prediction > 0.5))
        ).inc()

        return {"prediction": prediction}

    except Exception as e:
        inference_errors.labels(
            model_version=model_version,
            error_type=type(e).__name__
        ).inc()
        raise

    finally:
        active_requests.labels(model_version=model_version).dec()

Scrape these metrics with Prometheus, visualize with Grafana, and set up alerts:

# Prometheus alert rule
- alert: HighInferenceLatency
  expr: histogram_quantile(0.95, model_prediction_latency_seconds) > 0.1
  for: 5m
  annotations:
    summary: "p95 inference latency exceeds 100 ms"

- alert: HighErrorRate
  expr: rate(model_inference_errors_total[5m]) > 0.01
  for: 1m
  annotations:
    summary: "Model error rate exceeds 1%"

Architecture for Production ML Serving

A real production system looks like:

Client Requests
    |
    v
[API Gateway / Load Balancer]
    |
    v
[Model Server Pool]
  - Request batching
  - Model versioning (v1, v2, v3)
  - Concurrent inference on GPU
    |
    v
[Model Cache]
  - Cache recent predictions
  - Avoids redundant computation
    |
    v
[Monitoring / Logging]
  - Latency, errors, predictions
  - Alert triggers
  - Feedback collection
    |
    v
[Human Review / Fallback]
  - For high-stakes decisions
  - Manual override for appeals

Real serving frameworks (TorchServe, TensorFlow Serving, ONNX Runtime) handle batching, versioning, and health checks. FastAPI + Uvicorn is suitable for moderate traffic. At scale (millions of requests/day), you need specialized ML serving infrastructure.

SLA-Driven Architecture Patterns

Different SLA requirements call for different architectures:

| SLA Profile | Latency | Throughput | Example | Architecture | |-----------|---------|-----------|---------|--------------| | Interactive (p95 < 100ms) | Strict | 1k-10k req/s | Web API, search | Caching + batching with 20ms timeout | | Batch processing (p95 < 1s) | Loose | 100k+ req/s | Recommendation batch | Large batches, GPU optimized | | Streaming (p95 < 10ms) | Ultra-strict | 100-1k req/s | Real-time fraud | Single prediction, cached, no batching | | Best-effort (p95 < 5s) | Very loose | 10k+ req/s | Analytics pipeline | Maximum batching, cost-optimized |

Choose architecture to match your SLA, not the other way around.

Deployment Strategy Comparison

| Strategy | Rollout Time | Risk | Validation | Use Case | |----------|-------------|------|-----------|----------| | Big bang | Minutes | Very high | Minimal | Dev/test only | | Blue-green | Minutes | High | Medium (parallel run) | Low-traffic services | | Canary (5→25→100%) | Hours-days | Low | High (real traffic) | Production, high-stakes | | Shadow (parallel) | Days | Minimal | Very high (offline compare) | Fraud/lending, critical systems | | Feature flag | Minutes | Low (feature-gated) | High (controlled rollout) | Any critical path |

Recommended for ML models: Canary (5% traffic 1 hour → 25% → 100%) with automatic rollback if metrics degrade.

Caching and Request Deduplication

Beyond batching, two more optimizations dramatically reduce inference load:

Caching: If the same input arrives multiple times, reuse the cached prediction instead of re-running inference. Effective in recommendation systems, autocomplete, and image classification where duplicate requests are common.

from functools import lru_cache

class CachedModelServer:
    def __init__(self, model, cache_size=10000):
        self.model = model
        self.cache = {}
        self.cache_size = cache_size
        self.hits = 0
        self.misses = 0

    def predict(self, features_hash):
        """
        features_hash: a hashable representation of input features
        (e.g., tuple for small inputs, SHA256 for large inputs)
        """
        if features_hash in self.cache:
            self.hits += 1
            return self.cache[features_hash]

        # Cache miss; run inference
        self.misses += 1
        prediction = self.model.predict(features_hash)

        # Add to cache (with LRU eviction if full)
        if len(self.cache) >= self.cache_size:
            # Remove least-recently-used entry
            oldest_key = next(iter(self.cache))
            del self.cache[oldest_key]

        self.cache[features_hash] = prediction
        return prediction

    def hit_rate(self):
        total = self.hits + self.misses
        return self.hits / total if total > 0 else 0

In practice, cache hit rates of 50-80% are common. This is a huge win: a cache hit is ~1 ms (hash lookup), vs. 50 ms inference.

Request deduplication: During a batch window, if two requests have identical inputs, deduplicate them and run inference once. This is especially powerful for multi-user systems where many users make identical requests simultaneously (e.g., "predict sentiment of the same tweet").

# Batch with deduplication
request_queue = asyncio.Queue()
batch_size = 32
batch_timeout_ms = 20

async def deduplicating_batch_processor():
    while True:
        batch_inputs = {}  # {input_hash: [indices of requests with this input]}
        batch_futures = []
        deadline = time.time() + batch_timeout_ms / 1000

        # Collect unique requests
        while len(batch_inputs) < batch_size:
            wait_time = deadline - time.time()
            if wait_time <= 0:
                break

            try:
                future, input_features = await asyncio.wait_for(
                    request_queue.get(),
                    timeout=wait_time
                )
                input_hash = hash(tuple(input_features))

                if input_hash not in batch_inputs:
                    batch_inputs[input_hash] = []
                    batch_futures.append((future, input_features))

                batch_inputs[input_hash].append(future)

            except asyncio.TimeoutError:
                break

        if batch_futures:
            # Run inference only on unique inputs
            unique_inputs = torch.tensor(
                [inp[1] for inp in batch_futures],
                dtype=torch.float32,
                device=device
            )

            with torch.no_grad():
                predictions = model(unique_inputs)

            # Broadcast results to all requests with that input
            for i, (original_future, _) in enumerate(batch_futures):
                prediction = predictions[i]

                # Find all futures that share this input
                input_hash = hash(tuple(batch_futures[i][1]))
                for future in batch_inputs[input_hash]:
                    future.set_result(prediction)

If 10 users request recommendations for the same product ID simultaneously, you run inference once and return the same result 10 times. Massive throughput improvement.

Production Serving Frameworks

For production, don't build serving infrastructure from scratch. Use battle-tested frameworks:

TorchServe (PyTorch): Built by Meta/PyTorch team. Supports batching, versioning, A/B testing out of the box. Integrates with Kubernetes. Good for PyTorch models.

TensorFlow Serving (TensorFlow): Similar to TorchServe. Used extensively in industry. Excellent performance on CPU/GPU. Mature production deployments.

ONNX Runtime (Framework-agnostic): Inference engine that optimizes any model exported to ONNX format. Fastest for latency-critical applications. Good for model portability.

Seldon Core (Kubernetes-native): Open-source ML serving platform. Supports multiple frameworks, canary deployments, feedback loops. Best for complex ML pipelines.

vLLM, Ollama (LLM-specific): Specialized for large language models. Batching and KV cache optimization built-in.

Choose based on: framework (PyTorch vs. TensorFlow), deployment environment (Kubernetes, cloud-native, edge), and feature needs (A/B testing, monitoring, explainability).

Capacity Planning and Cost Analysis

Before deploying, understand your infrastructure requirements. Here's a framework:

Define your SLA:

• Throughput: X requests/second (peak, not average)
• Latency: p50 < Y ms, p95 < Z ms, p99 < W ms
• Availability: 99.95% uptime (max 22 minutes downtime/month)

Calculate hardware needs:

# Example: Fraud detection model
inference_latency_ms = 50  # Measured on your hardware
batch_size = 32  # Optimal batch size
batch_latency_ms = 100  # Measured with batch_size=32

# Throughput per GPU
batch_throughput = (batch_size / batch_latency_ms) * 1000  # requests/second
print(f"Throughput per GPU: {batch_throughput:.0f} req/s")

# Peak load
peak_requests_per_sec = 50000  # Your peak traffic

# GPUs needed
gpus_needed = peak_requests_per_sec / batch_throughput
print(f"GPUs needed: {gpus_needed:.0f}")

# Add 20% buffer for failover, maintenance
gpus_with_buffer = gpus_needed * 1.2
print(f"GPUs with buffer: {gpus_with_buffer:.0f}")

# Cost (illustrative estimate)
cost_per_gpu_per_month = 500  # NVIDIA A100 on cloud
monthly_cost = gpus_with_buffer * cost_per_gpu_per_month
print(f"Monthly cost: ${monthly_cost:,.0f}")

# Cost per prediction
predictions_per_month = peak_requests_per_sec * 86400 * 30  # Average case, not peak
cost_per_prediction = monthly_cost / predictions_per_month
print(f"Cost per prediction: ${cost_per_prediction:.6f}")

Trade-offs for cost optimization:

1. Batching: Increase batch size → better GPU utilization, lower cost/prediction, but higher p99 latency
2. Model compression (lesson 21): Smaller model uses cheaper GPU, or less GPU needed
3. Caching (this lesson): Reuse predictions → fewer inference calls → lower cost
4. Quantization (lesson 21): Integer math is faster → need fewer GPUs

Example cost scenarios (illustrative estimates):

• No optimization: 50,000 req/s, 100ms latency, 100 GPUs, $50k/month
• Batch size 128: Same throughput, 1ms added latency for early requests, 50 GPUs, $25k/month
• • Compression (4x): 25 GPUs, $12.5k/month (10ms latency still)
• • 50% cache hit rate: 12-13 GPUs, $6k/month, mostly cache hits

Auto-Scaling Policies

In production, traffic varies by time of day, seasonality, and events. Design auto-scaling:

# Example Kubernetes HPA (Horizontal Pod Autoscaler)
# Scales up if p95 latency exceeds SLA or GPU utilization > 80%

spec:
  minReplicas: 10  # Minimum pods for availability
  maxReplicas: 100  # Don't scale beyond this
  targetCPUUtilizationPercentage: 70  # Trigger scale-up at 70% CPU
  targetAverageValue: "50m"  # Target latency metric

# Scale-up policy: add 10 pods every 30 seconds
scaleUpBehavior:
  stabilizationWindowSeconds: 30
  policies:
  - type: Percent
    value: 100  # Double replicas
    periodSeconds: 30

# Scale-down policy: remove 1 pod every 5 minutes (conservative)
scaleDownBehavior:
  stabilizationWindowSeconds: 300
  policies:
  - type: Pods
    value: 1
    periodSeconds: 300

Monitor auto-scaling effectiveness:

• Track time to scale up (should be < 2 minutes to handle traffic spikes)
• Monitor cost variance (is auto-scaling preventing runaway costs?)
• Validate that SLA is met during scale-up (p95 latency shouldn't spike)

Load Testing and Capacity Planning Checklist

Before going to production, validate your system:

# Load test your serving system
import asyncio
import time
import statistics

async def load_test(num_requests=1000, concurrent=50):
    """
    Simulate production load.
    """
    latencies = []
    errors = 0

    async def make_request(request_id):
        try:
            start = time.time()
            response = await predict({"features": [1.0, 2.0, 3.0]})
            latency = (time.time() - start) * 1000  # ms
            latencies.append(latency)
            return True
        except Exception as e:
            print(f"Error in request {request_id}: {e}")
            return False

    # Send requests concurrently
    tasks = [make_request(i) for i in range(num_requests)]
    results = await asyncio.gather(*tasks)
    errors = sum(1 for r in results if not r)

    # Analyze results
    latencies_sorted = sorted(latencies)
    print(f"=== Load Test Results ({num_requests} requests, {concurrent} concurrent) ===")
    print(f"Success rate: {(len(latencies) / num_requests) * 100:.1f}%")
    print(f"p50 latency: {statistics.median(latencies):.2f}ms")
    print(f"p95 latency: {latencies_sorted[int(len(latencies) * 0.95)]:.2f}ms")
    print(f"p99 latency: {latencies_sorted[int(len(latencies) * 0.99)]:.2f}ms")
    print(f"Max latency: {max(latencies):.2f}ms")
    print(f"Mean latency: {statistics.mean(latencies):.2f}ms")

    # Validate SLA
    sla_p95 = 100  # ms
    actual_p95 = latencies_sorted[int(len(latencies) * 0.95)]
    if actual_p95 > sla_p95:
        print(f"✗ SLA VIOLATED: p95={actual_p95:.0f}ms > target={sla_p95}ms")
    else:
        print(f"✓ SLA MET: p95={actual_p95:.0f}ms <= target={sla_p95}ms")

# Run load test
asyncio.run(load_test(num_requests=1000, concurrent=50))

Common Mistake: Optimizing for Average Latency Instead of Percentiles

Your SLA says p95 latency must be < 100 ms. You optimize your system and achieve p50 latency of 30 ms and p99 of 150 ms. Your average is 50 ms. Looks good?

No. Your SLA is violated 5% of the time. Some customers experience 150 ms delays repeatedly. You must optimize for percentiles, not averages.

This often happens with batching: if batch timeout is too high, some requests pile up and experience long waits. Set timeout low enough that p95 latency stays under your SLA.

# Track percentiles, not just average
import numpy as np

latencies = []

for _ in range(1000):
    latencies.append(inference_latency())

latencies = np.array(latencies)

print(f"p50: {np.percentile(latencies, 50):.2f} ms")
print(f"p95: {np.percentile(latencies, 95):.2f} ms")
print(f"p99: {np.percentile(latencies, 99):.2f} ms")
print(f"average: {np.mean(latencies):.2f} ms")
print(f"max: {np.max(latencies):.2f} ms")

# Use p95 (or whatever percentile your SLA specifies) for compliance
# Don't rely on average

SLAs are about percentiles, not averages. Design your system accordingly. Set batch timeouts, buffer sizes, and scaling thresholds to keep p95/p99 under budget, not average.