How LLM inference serving works: prefill, decode, continuous batching and throughput

Key points

• Prefill is compute-bound and sets the time to first token; decode is memory-bandwidth-bound and sets the inter-token latency.
• Because every decoding step reads all the weights once, batching more sequences into that step raises throughput almost for free until compute or KV cache memory runs out.
• Static and dynamic batching schedule whole requests, so short answers wait for long ones. Continuous batching, from the Orca paper, schedules one iteration at a time.
• Chunked prefill splits long prompts so they stop stalling running decodes; disaggregation moves prefill and decode onto separate GPUs.
• Speculative decoding lets a cheap draft propose several tokens that the target model verifies in one pass, without changing the output distribution.
• Load-test with realistic prompt and output lengths, sweep the load, and report goodput: the requests that met their latency targets, not raw tokens per second.

Prefill and decode: two different workloads

A decoder-only language model generates text one token at a time, and each new token depends on every token before it. Serving a request therefore happens in two phases:

• Prefill: the server runs the whole prompt through the model in one forward pass. All prompt tokens are known up front, so the work is large matrix-matrix multiplication that keeps the GPU's arithmetic units busy. Prefill fills the KV cache with the keys and values of every prompt token and produces the first output token.
• Decode: the server generates the remaining tokens one per forward pass. Each pass processes a single new token per sequence, reads the KV cache for everything before it, and appends one more key and value.

The two phases stress different parts of the hardware. NVIDIA's guide to inference optimization describes prefill as a highly parallel matrix-matrix operation that saturates the GPU, and decode as a memory-bound operation that leaves much of its compute idle. The reason is simple arithmetic. To produce one token, a decoding step has to read every weight of the model from GPU memory once, plus that sequence's KV cache, but it does only a few operations per byte it reads.

An illustrative estimate: a 7-billion-parameter model stored in 16-bit precision is about 14 GB of weights. On a GPU that reads memory at about 2 TB per second, reading those weights once takes about 7 milliseconds, so one sequence on its own cannot decode much faster than about 140 tokens per second, however much compute sits idle. Put 32 sequences into the same step and the weights are still read once, but they now produce 32 tokens. That is the whole case for batching: decode throughput grows with batch size, nearly linearly at first, until the step becomes compute-bound or the KV caches of all those sequences no longer fit in memory.

Most serving techniques follow from this split. Batching fills idle compute during decode; KV cache management decides how big the batch can get; chunked prefill and disaggregation stop the two phases from getting in each other's way; speculative decoding and quantization cut the memory traffic per useful token. For what the KV cache holds and how big it gets, see prompt caching and the KV cache.

The metrics: TTFT, inter-token latency, throughput and goodput

NVIDIA's benchmarking documentation defines TTFT as the time from query submission to the first received token, and computes ITL as end-to-end latency minus TTFT, divided by the number of output tokens minus one, so that it describes only the decoding part. It also states the trade-off every serving stack faces: as concurrent requests grow, total tokens per second rises until the GPU saturates, while tokens per second for each user falls.

That trade-off is why raw throughput is a poor target on its own. A server can always raise tokens per second by packing in more sequences, at the price of slower streams and longer queues. The DistServe paper (Zhong and colleagues, OSDI 2024) frames the goal as goodput: the maximum request rate that can be served while meeting both a TTFT target and a TPOT target. A chat product cares most about TTFT and a steady ITL; an offline batch job cares only about throughput.

Measured from the client, TTFT includes the network, the queue and prefill, which is what a user feels. A sketch that times any token stream:

import statistics, time

def measure(stream):
    start = time.perf_counter()
    stamps = [time.perf_counter() for _ in stream]  # a timestamp as each token arrives
    gaps = [b - a for a, b in zip(stamps, stamps[1:])]
    return {
        "ttft_s": stamps[0] - start,
        "itl_mean_s": (stamps[-1] - stamps[0]) / (len(stamps) - 1),
        "itl_p99_s": statistics.quantiles(gaps, n=100)[98],
        "e2e_s": stamps[-1] - start,
        "output_tokens": len(stamps),
    }

def fake_stream(prefill_s=0.25, step_s=0.02, n=40):  # stands in for a streaming client
    time.sleep(prefill_s)
    for i in range(n):
        if i:
            time.sleep(step_s)
        yield f"token-{i}"

print({k: round(v, 2) for k, v in measure(fake_stream()).items()})
# one run (timings vary a little): {'ttft_s': 0.25, 'itl_mean_s': 0.02, 'itl_p99_s': 0.03, 'e2e_s': 1.17, 'output_tokens': 40}

With a real server, fake_stream becomes the iterator over a streaming response. Two cautions: a streamed chunk can carry more than one token, so count tokens with the server's usage figures or the model's tokenizer rather than counting chunks, and report percentiles, because a mean ITL hides the stalls users notice. For tracing these numbers per request in a live system, see LLM observability and tracing; for how tokens reach a browser, see WebSockets vs server-sent events.

Static, dynamic and continuous batching

Static batching groups a fixed set of requests and runs them together until every one has finished. Requests finish at different times, so a batch that holds one long answer keeps its slots occupied by sequences that are already done, and new requests wait for the whole batch.

Dynamic batching, as general-purpose model servers implement it, forms each batch from whatever is queued: it combines requests in arrival order up to a maximum batch size, and holds them for at most a configured delay while it waits for more. That helps models whose requests take one forward pass each. For text generation it still schedules whole requests: once a batch starts, its membership is fixed until it ends.

Continuous batching changes the unit of scheduling from the request to the iteration. The Orca paper (Yu and colleagues, OSDI 2022) calls this iteration-level scheduling: after every forward pass, the scheduler returns finished sequences to their clients and admits waiting ones into the freed slots. Sequences in one batch are then at different positions, with different lengths, so Orca adds selective batching: operations such as the linear layers run batched across all the tokens of all the sequences, while attention, which depends on each sequence's own history, runs per sequence. The authors report up to 36.9 times the throughput of NVIDIA FasterTransformer at the same latency on a 175-billion-parameter GPT-3 model. NVIDIA's guide calls the same idea in-flight batching, and it is now the standard way LLM servers schedule decoding.

A toy simulation shows where the gain comes from. It assumes every decoding step takes the same time whatever the batch size, which is roughly true while decode is memory-bound, ignores prefill, and starts with 1,000 queued requests whose output lengths vary from 10 to 500 tokens:

import random

def simulate(lengths, slots, continuous):
    queue, active, steps, tokens = list(lengths), [], 0, 0
    while queue or active:
        if continuous or not active:  # static batching refills only an empty batch
            while queue and len(active) < slots:
                active.append(queue.pop(0))
        steps += 1                    # one decode step: one token per running sequence
        tokens += len(active)
        active = [left - 1 for left in active if left > 1]
    return steps, tokens / steps

random.seed(7)
lengths = [random.randint(10, 500) for _ in range(1000)]  # output tokens per request
for slots in (8, 32):
    s_steps, s_rate = simulate(lengths, slots, continuous=False)
    c_steps, c_rate = simulate(lengths, slots, continuous=True)
    print(f"slots={slots}: static {s_steps} steps, {s_rate:.1f} tok/step; "
          f"continuous {c_steps} steps, {c_rate:.1f} tok/step")
# slots=8: static 55128 steps, 4.5 tok/step; continuous 31111 steps, 8.0 tok/step
# slots=32: static 15492 steps, 16.0 tok/step; continuous 8019 steps, 30.9 tok/step

In this run, continuous batching finishes the same work in 31,111 steps instead of 55,128 with 8 slots, and 8,019 instead of 15,492 with 32 slots: about 1.8 and 1.9 times the throughput, with the slots almost always full. The size of the gain depends on how much output lengths vary. If every request produced exactly the same number of tokens, the two schedulers would tie. Real traffic, where one answer is a word and the next is a page, is the case continuous batching was built for.

KV cache memory, preemption and chunked prefill

Continuous batching moves the limit from scheduling to memory. Every running sequence holds a KV cache that grows by one entry per token, and the batch can only be as large as the caches that fit beside the weights. PagedAttention, introduced with vLLM, stores each cache in fixed-size blocks found through a block table, so memory is allocated as sequences grow rather than reserved for their maximum length. The details, and how the same blocks enable prefix caching, are in prompt caching and the KV cache.

When memory runs out anyway, a server has to preempt. vLLM's optimization guide says it preempts requests when KV cache space is short and recomputes them later, which can hurt end-to-end latency, and suggests giving the cache more GPU memory, lowering the maximum number of concurrent sequences or batched tokens, or adding tensor or pipeline parallelism to free memory.

The second problem is interference. When a long prompt arrives, its prefill joins a batch whose other members are decoding. That step now takes as long as the prefill, so every running stream pauses: a visible stall in the inter-token latency. Sarathi-Serve (Agrawal and colleagues, OSDI 2024) attacks this with chunked prefill: a long prompt is split into chunks of roughly equal size, and each step carries one chunk alongside the ongoing decodes, under a fixed token budget. Its stall-free scheduling admits new requests without pausing running decodes. The authors report up to 2.6 times higher serving capacity than vLLM for Mistral-7B on one A100 GPU under tail-latency targets, and larger gains on bigger models.

vLLM now enables chunked prefill by default where it can. Its scheduler batches all pending decodes before scheduling any prefill, fills the remaining budget set by max_num_batched_tokens with prefill work, and chunks prefills that do not fit. The documentation spells out the trade-off: a smaller budget favors inter-token latency, because less prefill slows each step, and a larger one favors time to first token, because more prompt tokens get processed per step.

Prefill and decode disaggregation

Chunking limits the interference but keeps both phases on the same GPUs, with the same parallelism. DistServe goes further: it runs prefill and decode on different GPUs. A prefill instance processes the prompt and hands the KV cache to a decode instance, which generates the rest. Each pool can then be sized and parallelized for its own phase and its own latency target, rather than one configuration compromising between them. The authors report serving up to 7.4 times more requests, or meeting 12.6 times tighter latency targets, than the systems they compared against, with more than 90 percent of requests within their latency constraints.

The trade-off is moving the KV cache between machines on every request. That needs fast interconnect, placement that keeps the transfer short, and two pools to size and operate instead of one. Disaggregation pays off at scale, with steady traffic and tight targets on both TTFT and ITL; a single-node deployment usually gets most of the way with chunked prefill.

Speculative decoding

Batching raises throughput across users, but it does nothing for one stream on a quiet server: each token still costs a full read of the weights. Speculative decoding attacks that. A cheap drafter proposes the next few tokens, and the large target model checks all of them in a single forward pass, which costs about the same as generating one token because decode is memory-bound. The target keeps the longest prefix of the draft it agrees with, plus one token of its own, so every step yields at least one token and often several.

Two groups published the method independently. Leviathan, Kalman and Matias (ICML 2023) call it speculative decoding and report two to three times faster generation on T5-XXL with identical outputs. Chen and colleagues at DeepMind call it speculative sampling and report a 2 to 2.5 times decoding speedup on the 70-billion-parameter Chinchilla. Both use a modified rejection sampling rule that accepts or replaces draft tokens so the final output follows the target model's distribution exactly, up to hardware numerics. It is an optimization, not an approximation.

The drafter can be a smaller model from the same family, extra prediction heads trained on the target, or no model at all: n-gram lookup copies likely continuations from the prompt, which works well for editing and extraction tasks that repeat their input. vLLM lists several such methods, says its implementation is lossless up to hardware precision apart from one variant that relaxes verification on purpose, and describes the goal as reducing inter-token latency in medium-to-low load, memory-bound workloads. That is the main caveat: when the batch is already large and the GPU is compute-bound, verifying rejected drafts is wasted work, and the speedup shrinks or reverses. Gains also depend on the acceptance rate, so measure on your own prompts.

Quantization, tensor parallelism and pipeline parallelism

Quantization stores weights, and sometimes activations and the KV cache, in fewer bits: 8-bit integers or 8-bit floating point instead of 16-bit, or 4-bit weights. Since decode is bound by memory traffic, halving the bytes per weight roughly halves the data each step has to read, and it frees memory for more KV cache and so a larger batch. NVIDIA's guide notes that most deep learning models can be represented with eight or fewer bits per value. The cost is accuracy, which varies by method, model and task, so compare the quantized model with the original on your own evaluation set before switching.

When a model does not fit on one GPU, it has to be split. Tensor parallelism splits each layer's weight matrices across GPUs, so every GPU works on every token and they exchange partial results inside each layer; it needs very fast links and is the usual choice within one node, and it also leaves each GPU more room for KV cache. Pipeline parallelism gives each GPU a contiguous slice of layers and passes activations from one stage to the next; it needs less bandwidth and suits splitting across nodes, but stages can sit idle waiting for each other, the pipeline bubbles NVIDIA describes. vLLM's guide recommends tensor parallelism first and pipeline parallelism once tensor parallelism is exhausted or the model spans nodes, and the two can be combined.

How to load-test a serving stack

A serving benchmark is only as good as its workload. Throughput at 128 input tokens and 128 output tokens tells you little about a RAG assistant that sends 6,000-token prompts. A workable procedure:

1. Take the distributions of prompt length, output length and shared prefixes from real traffic or logs, and decide whether clients stream.
2. Set the targets first: for example a p99 TTFT and a p99 TPOT for interactive traffic, or none for offline jobs. Goodput is measured against them.
3. Warm the server up and discard the first requests, which include compilation, memory allocation and cold caches.
4. Sweep the load upward: fixed concurrency levels to find the saturation point, and an open-loop request rate, with Poisson arrivals, to see how queueing behaves at a given rate. A closed loop alone hides queueing, because clients wait before sending more.
5. At each level record TTFT and ITL at p50 and p99, output tokens per second, completed requests per second, goodput, and errors or overload responses.
6. Find the knee: the load beyond which p99 TTFT climbs steeply while throughput flattens. Capacity planning lives just below it.
7. Change one setting at a time, such as the batched-token budget, maximum concurrent sequences, chunked prefill, parallelism, quantization or speculative decoding, and rerun the same sweep.

# Concurrency sweep against an OpenAI-compatible server; vLLM's benchmark CLI as one example
for c in 1 4 16 64 128; do
  vllm bench serve --base-url http://localhost:8000 --model my-model \
    --dataset-name random --random-input-len 2000 --random-output-len 300 \
    --num-prompts 500 --max-concurrency "$c" \
    --percentile-metrics ttft,tpot,itl,e2el --metric-percentiles 50,99 \
    --goodput ttft:1000 tpot:50 --save-result
done
# Plot output tokens per second and p99 TTFT against concurrency; find the knee.

vLLM's vllm bench serve is one example of such a tool: it sends requests at a set rate with Poisson or burstier arrivals, caps concurrency, generates random prompts of set lengths, reports TTFT, TPOT, ITL and end-to-end percentiles, and counts goodput against ttft, tpot and e2el targets in milliseconds. Whatever tool you use, check that the load generator is not the bottleneck, test once through the full gateway and network path that users hit, and fix output lengths when comparing configurations, then confirm the result with realistic stopping. Rate limits in front of the server shape what it sees; see rate limiter system design. For the cost side of the same choices, see how to reduce LLM costs.

If you are building an AI agent or assistant and need it to stay fast under real traffic, see our AI agent development service: the code lives in your GitHub from day one, and AI-assisted work is reviewed by an engineer.