Serving LLMs Efficiently

One request, two machines

Serving a large language model is unlike serving any other model, and nearly every counterintuitive thing about its performance follows from a single fact: a single request runs in two phases with opposite bottlenecks.

Prefill processes the entire prompt. Every token is available at once, so the work happens in parallel across the sequence — large matrix multiplications that saturate an accelerator. Prefill is compute-bound. Its cost scales with prompt length, and it determines time-to-first-token, the delay before anything appears on screen.

Decode generates the output, one token at a time. Each new token depends on all previous ones, so there is no parallelism across the sequence. Each step performs relatively little arithmetic while reading the entire model weights and the accumulated attention cache from memory. Decode is memory-bandwidth-bound. Its cost scales with output length, and it determines the tokens-per-second rate a user perceives as typing speed.

The practical consequences are worth stating explicitly, because they surprise people:

A single decode step barely uses the accelerator's compute. You are paying for a device that spends most of its time waiting on memory. This is why batching helps so dramatically here — the weights are read once and applied to many sequences at once, so more concurrent requests are nearly free in compute terms.

Optimizations do not transfer between phases. Something that halves prefill time may do nothing for decode. Report and track time-to-first-token and inter-token latency as separate metrics; a single average latency number blends two unrelated bottlenecks into one uninformative figure.

A summarization workload and a chat workload are different systems. Long prompt, short output is prefill-heavy. Short prompt, long output is decode-heavy. Running both through one deployment tuned one way serves one of them badly.

The KV cache is your real capacity limit

During decode, the model must attend to every previous token. Recomputing their key and value projections at every step would be enormously wasteful, so they are cached — the KV cache.

That cache is the hidden constraint on how many requests you can serve at once. It grows linearly with sequence length and with the number of concurrent sequences, and it lives in the same accelerator memory as the model weights. The arithmetic is unforgiving: with weights occupying most of the device, what remains sets a hard ceiling on concurrency, and a handful of long-context conversations can consume what a much larger number of short ones would.

Three consequences follow, and they explain most confusing production behavior:

Long contexts cost memory, not just compute. Doubling context length roughly doubles per-request cache. Concurrency drops accordingly. This is why raising a context limit "just to be safe" can halve your throughput without any change in traffic.

Naive allocation wastes most of the cache. A serving system that reserves a contiguous block for each sequence's maximum possible length wastes everything between the actual and maximum length. This was the dominant inefficiency in early LLM serving, and it is what paged attention addresses — allocating the cache in small blocks, the way an operating system pages memory, so allocation tracks actual use rather than worst-case reservation. The published work reports large throughput gains from this alone, and it is now standard in modern serving stacks.

Cache is a resource to schedule, not a detail. Systems that treat available cache blocks as a scheduling input can admit as many requests as genuinely fit and preempt when necessary. Systems that do not will either over-admit and fail, or under-admit and idle.

The operational takeaway: if you are choosing a serving stack, the memory management strategy matters more than raw kernel speed. And if your throughput is disappointing, check cache utilization before adding hardware — the most common finding is a fragmentation or over-reservation problem, not a capacity problem.

Static versus continuous batching

Batching for an LLM is unlike batching for any other model, because sequences in a batch finish at wildly different times.

Static batching collects requests, runs them together, and returns when all are done. For a classifier, where every input takes the same time, this is fine. For generation it is close to pathological: one request producing thirty tokens and another producing eight hundred are in the same batch, and the short one's slot sits occupied and idle for the entire remaining duration. With realistic length variation, most of your paid accelerator time is spent on finished sequences waiting for their batch mates.

Continuous batching — also called iteration-level scheduling — changes the unit of scheduling from the batch to the token step. After each step, finished sequences leave and queued requests take their slots immediately. The batch is a rolling window rather than a fixed group.

The improvement is not marginal. Under variable output lengths, which describes every real workload, continuous batching commonly delivers several times the throughput of static batching on identical hardware. It is the single largest lever in LLM serving, and unlike quantization it costs nothing in output quality — the arithmetic is unchanged, only the scheduling differs.

This is why you should reach for a purpose-built serving stack rather than wrapping a generation call in a web framework. A naive implementation processes one request at a time or batches statically, and leaves most of the hardware idle. The gap between a naive server and a modern one on the same GPU is large enough to change what your product costs to run.

Two knobs are worth understanding once you are on such a stack:

Maximum concurrent sequences. Bounded by KV cache memory. Set too high, requests are preempted and latency becomes erratic. Set too low, the device idles.

Scheduling policy between prefill and decode. Admitting a new request means running its prefill, which is compute-heavy and briefly stalls decoding for everyone else. Aggressive admission gives good time-to-first-token for new arrivals and choppy output for existing ones. Chunked prefill — splitting a long prompt across several steps — smooths this considerably and is worth enabling for mixed workloads.

Choosing a batch policy from the workload

Throughput and latency are one dial, and there is no setting that is good for everything.

Small batches. Low latency per request, poor device utilization, high cost per token. Correct when a human is watching tokens appear and the perceived speed is the product.

Medium batches. The balance most production APIs want. Good utilization with a bounded queue wait. Enforce a ceiling on queue time so a request never waits indefinitely for a batch to fill.

Large batches. Maximum throughput, long queue waits, lowest cost per token. Correct when nobody is waiting — bulk classification, offline generation, embedding backfills.

The mistake is running all workloads through one deployment with one setting. An interactive chat feature and a nightly document-processing job have opposite requirements, and a single compromise serves both poorly: chat feels sluggish and the batch job costs more than it should.

Separate the pools. Interactive traffic gets a small-batch, latency-optimized deployment. Bulk traffic gets a large-batch, throughput-optimized one, ideally on cheaper interruptible capacity since nothing is waiting on it. The two can share model weights and infrastructure patterns while having entirely different scheduling configuration.

When capacity is still short after the pools are separated, the remaining levers change the model rather than the schedule. Quantization to lower precision reduces both weight memory and bandwidth pressure, which helps decode specifically and frees cache memory for concurrency. Speculative decoding uses a small draft model to propose tokens that the large model verifies in parallel, converting some memory-bound decode steps into compute-bound verification. Both are real wins and both change output, so both need evaluation before release — a quantized model is a new model and deserves the same promotion gate as any other candidate.

Worked example: reading a bad latency report

A team reports that their LLM feature has "800 ms average latency" and wants a bigger GPU. Three questions dismantle the report.

Which phase? Splitting the metric shows time-to-first-token at 120 ms and total generation at 800 ms for an average of 200 output tokens. That is 3.4 ms per token, which is reasonable. The feature does not have a latency problem; it has an output-length problem. The prompt asks for a detailed explanation and the product only displays the first two sentences. Constraining output length in the prompt and the sampling parameters cuts perceived latency by more than half with no infrastructure change at all.

What is the utilization? Twelve percent, with an average of 1.4 concurrent sequences. The GPU is almost entirely idle. Investigation finds the application serializes requests behind a lock added months earlier to fix an unrelated race. Removing it raises concurrency to 20 and throughput by an order of magnitude, with per-request latency essentially unchanged — the expected result when decode is memory-bound and the device had spare bandwidth.

Are the workloads mixed? Yes. The same endpoint serves an interactive assistant and a nightly summarization job over long documents. The summarization prefills are long and compute-heavy, and when one is admitted, interactive users see a visible stall. Splitting into two pools removes the interference. The summarization pool then moves to larger batches and spot capacity, cutting its cost substantially.

Final state: no new hardware. Latency improved, throughput improved by roughly an order of magnitude, and cost went down. The original request — a bigger GPU — would have addressed none of the three actual problems, and the twelve percent utilization figure was sitting there the whole time.

The general lesson is the same one from serving architectures generally, sharpened: for LLM serving, the first three numbers to collect are time-to-first-token, inter-token latency, and device utilization. They almost always identify the real constraint, and it is rarely the one in the ticket.

Common mistake

The most common mistake is reporting a single average latency. It merges a compute-bound phase and a memory-bound phase into one number that cannot guide any decision. A model with fast prefill and slow decode and a model with the reverse can report identical averages and need completely different fixes.

The second mistake is scaling out before checking utilization. Adding replicas to a deployment sitting at twelve percent utilization multiplies the cost of idleness. Low utilization on an LLM deployment nearly always means a batching, concurrency, or application-level serialization problem, and every one of those is cheaper to fix than to outspend.

The correcting habit: before requesting hardware, produce three numbers — time-to-first-token at p99, inter-token latency at p99, and average device utilization. If utilization is low, the problem is upstream of capacity. If time-to-first-token is high, look at prompt length and admission scheduling. If inter-token latency is high, look at batch size and memory bandwidth. Each number points at a different fix, and none of them points at a bigger GPU as often as people expect.