- One Server, Three People, Three Very Different Requests
- Short Question
- Active Generation
- Long Document
- Why a Long Prompt Is Different From a Long Answer
- What Happens When a Large Prefill Arrives During Generation?
- The GPU Is Sharing More Than Arithmetic
- Chunked Prefill Changes How Long Prompts Enter the GPU
- Why Smaller Chunks Are Not Always Better
- Four Metrics Reveal Whether the Server Is Actually Responsive
- How to Investigate Long-Prompt Interference
- Establish a Short-Prompt Baseline
- Introduce Long Prompts
- Inspect Scheduling and Memory Pressure
- Compare Chunked-Prefill Configurations
- Measure the Trade-Off
- Repeat Under Realistic Load
- What Can You Actually Configure in vLLM?
- What If Chunked Prefill Is Not Enough?
- One GPU Does Not Have to Serve Every Workload the Same Way
- The Real Question Is Not Whether the GPU Is Busy
- A Faster AI Server Is Sometimes a Better-Scheduled AI Server
The AI assistant is working perfectly until someone uploads a large document. One employee asks a short question and waits. Another sends a simple follow-up and waits too. The GPU is busy, the server has not crashed, and nothing obvious appears broken. Yet the service suddenly feels slower for everyone.
This is one of the more frustrating problems in shared AI infrastructure. A server can have enough VRAM to hold its model, enough compute to generate responses quickly, and still deliver an uneven experience when requests of very different sizes arrive together.
The reason is that an LLM server does not perform the same amount of work for every request. Before producing an answer, it must process the incoming prompt. A short question and a long technical document place very different demands on the GPU.
When those requests share one inference engine, the scheduler must decide how much computation each receives and when. That decision can affect how quickly every user sees the first word of an answer.
The interesting part is that the problem is not necessarily a shortage of hardware. Sometimes it is how the existing hardware is scheduled.
- A long prompt can create substantial prefill work before its first output token appears.
- On a shared server, prefill and ongoing token generation compete for GPU execution time.
- Chunked prefill allows a scheduler to divide long prompt processing across multiple scheduling iterations.
- Better aggregate throughput does not guarantee that every user experiences better latency.
- Queue time, TTFT, inter-token latency, and KV-cache pressure should be measured separately.
One Server, Three People, Three Very Different Requests
Imagine a small company running an internal AI assistant on one GPU server.
The first employee asks the model to rewrite a short email. The second is already receiving a long explanation about a software problem. The third uploads a large contract and asks for a summary.
From the user’s perspective, these are simply three conversations. From the server’s perspective, they are three workloads with different computational shapes.
The email request contains relatively little input. The ongoing explanation requires repeated generation steps. The contract request may contain thousands of input tokens that must be processed before its answer can begin.
The server needs to share its GPU across all three without making the short requests wait unnecessarily or interrupting the longer response too severely.
Short Question
Small input, relatively little prefill work, and a short expected response.
Active Generation
The model is already producing output tokens and needs repeated decode steps.
Long Document
A large input creates substantial prompt-processing work before generation starts.
A good scheduler has to handle all three types of work. Simply keeping the GPU busy is not enough. It also needs to decide how to share execution capacity without creating unacceptable delays.
Why a Long Prompt Is Different From a Long Answer
An LLM inference request generally has two important phases: prefill and decode.
During prefill, the model processes the input prompt and builds the attention state needed for subsequent generation. A longer prompt generally means more input-token computation and more KV-cache state, although the exact cost depends on model architecture and implementation.
During decode, the model produces new tokens autoregressively. Each step uses the existing context and generates the next output token.
These phases place different demands on the GPU.
Prefill can process many input tokens in parallel and often creates a relatively large burst of computation. Decode repeatedly performs smaller steps, reading model weights and attention state as generation continues.
This distinction is why an inference server cannot be understood through one tokens-per-second number.
For a deeper explanation of this distinction, see our guide to continuous batching in LLM inference.
What Happens When a Large Prefill Arrives During Generation?
Suppose several requests are already generating responses. The server is performing repeated decode steps and streaming results to users.
A new request arrives with a much larger prompt.
The scheduler now has additional prefill work to perform. Depending on its scheduling policy, token budget, and available KV-cache capacity, that work may be admitted immediately, divided into smaller pieces, or left waiting.
If a large prefill occupies a substantial scheduling iteration, existing decode requests may experience longer gaps before their next tokens are produced.
The result can feel strange. The person who uploaded the document expects a delay, but someone asking a short question may also notice that the service has become less responsive.
This is a form of interference between workloads sharing the same accelerator.
The GPU Is Sharing More Than Arithmetic
Requests compete for GPU execution time, memory bandwidth, and available attention-state storage.
Longer prompts can increase the amount of prefill computation and the KV-cache memory needed for their contexts. As active sequences accumulate, the server must manage both compute scheduling and memory capacity.
The GPU may remain highly utilized throughout this process. High utilization, however, does not tell you whether requests are being served fairly or whether users are waiting too long.
Chunked Prefill Changes How Long Prompts Enter the GPU
One approach to this problem is called chunked prefill.
Instead of requiring a long prompt to be processed as one large scheduling unit, the inference engine can divide its prefill work into smaller portions.
Those portions can be scheduled across multiple iterations alongside other work, subject to the engine’s scheduling rules and available token budget.
This gives the scheduler more flexibility. It can make progress on a large prompt without necessarily allocating one oversized iteration to that request.
In vLLM, chunked prefill is connected to the scheduler’s token budget. The max_num_batched_tokens setting limits how many tokens can be scheduled within an iteration.
This does not mean every user automatically becomes faster. Smaller chunks can change scheduling overhead and the balance between prompt-processing efficiency and decode responsiveness.
The objective is to find a useful balance for the actual workload.
| Behavior | Large Prefill Scheduling | Chunked Prefill |
|---|---|---|
| Long prompt processing | May occupy a large scheduling unit. | Can be distributed across multiple iterations. |
| Interaction with decode | A large prefill iteration may increase gaps between output tokens. | Smaller prefill portions give the scheduler more opportunities to mix work. |
| Scheduling flexibility | Less granular when large prompts are admitted as whole units. | More granular allocation of the token budget. |
| Possible trade-off | Large prompt-processing batches may offer efficient execution. | More scheduling iterations can introduce overhead or change prefill efficiency. |
| Best evaluation | Measure prefill latency, TTFT, and decode responsiveness. | Measure the same metrics under identical concurrent workloads. |
The table describes scheduling trade-offs, not guaranteed benchmark outcomes. Actual behavior depends on the inference engine, model, GPU, request distribution, and configuration.
Why Smaller Chunks Are Not Always Better
It is tempting to assume that the smallest possible prefill chunks will produce the most responsive server.
But scheduling has a cost.
A large prefill can benefit from processing many input tokens together. Dividing the work more aggressively may create additional scheduling iterations and change GPU execution efficiency.
Meanwhile, a larger token budget may improve how much work the server completes per iteration, but it can also allow larger prefill workloads to influence the duration of individual steps.
The right setting therefore depends on what the server is trying to optimize.
A batch-oriented document-processing system may care primarily about total throughput. An interactive assistant serving employees may place greater value on consistent first-token latency and smooth streaming.
Both systems can use the same model and GPU while needing different scheduling policies.
Chunked prefill is not a universal fix for overloaded inference servers. If the GPU lacks sufficient compute capacity, the KV cache is under heavy pressure, or requests arrive faster than the system can complete them, scheduling changes alone cannot remove the underlying capacity limit.
Four Metrics Reveal Whether the Server Is Actually Responsive
A server can report impressive aggregate throughput while some users experience frustrating delays.
To understand that difference, separate the measurements that describe individual requests from those that describe the entire service.
| Metric | What It Measures | Why It Matters |
|---|---|---|
| Queue Time | Time spent waiting before execution is admitted. | Shows whether requests are accumulating faster than scheduling capacity allows. |
| TTFT | Time from request arrival to the first generated token. | Captures the delay users experience before a response begins. |
| Inter-Token Latency | Time between successive generated tokens. | Shows whether an active response streams smoothly or experiences interruptions. |
| Throughput | Aggregate work completed over time. | Shows overall serving capacity but does not describe fairness or individual latency by itself. |
| KV-Cache Usage | How much attention-state capacity is occupied. | Helps identify memory pressure that can restrict concurrency or trigger preemption. |
vLLM exposes production metrics for these different dimensions, including waiting requests, running requests, time to first token, inter-token latency, queue time, and KV-cache usage.
The most useful view is often not the average. Tail latency, such as the 95th or 99th percentile, can reveal a smaller group of requests that suffer disproportionately under load.
For an interactive assistant, those slow requests may be exactly what users remember.
How to Investigate Long-Prompt Interference
You do not need to guess whether long prompts are affecting other users. A controlled workload can make the effect visible.
The goal is not to invent an impressive benchmark number. It is to compare server behavior under repeatable conditions.
Establish a Short-Prompt Baseline
Run a representative set of short interactive requests at a fixed concurrency level. Record TTFT, inter-token latency, throughput, and queue time.
Introduce Long Prompts
Add requests containing longer documents while preserving the original short-request workload. Observe whether short-request latency changes.
Inspect Scheduling and Memory Pressure
Monitor waiting requests, running requests, GPU memory consumption, KV-cache utilization, and any reported preemption activity.
Compare Chunked-Prefill Configurations
Where supported, test different scheduler token budgets. Keep the model, GPU, prompt distribution, output lengths, and arrival pattern consistent.
Measure the Trade-Off
Check whether improved short-request responsiveness comes with changes in long-request completion time or total throughput.
Repeat Under Realistic Load
Use a request distribution that resembles production. A synthetic test with only one prompt length can hide the interference you are trying to understand.
What Can You Actually Configure in vLLM?
vLLM provides scheduler options for controlling how requests are admitted and how much token-processing work can be scheduled per iteration.
The relevant controls include chunked prefill, the maximum batched-token budget, and the maximum number of sequences.
For example, the following command illustrates how these options can be supplied to a vLLM server:
vllm serve Qwen/Qwen3-0.6B \
--enable-chunked-prefill \
--max-num-batched-tokens 2048 \
--max-num-seqs 32These values are illustrative, not recommended universal defaults. Their suitability depends on the model, hardware, context lengths, and workload. Check the options supported by your installed vLLM version before applying them to production.
A smaller batched-token budget may limit the amount of prefill work scheduled in an iteration. A larger budget may allow more work to be grouped together.
Neither direction guarantees better results across every workload.
Configuration changes should be evaluated against the latency targets that matter to your users.
What If Chunked Prefill Is Not Enough?
Sometimes the scheduler is behaving reasonably and the system is simply being asked to do more work than one GPU can comfortably support.
In that situation, several architectural choices become relevant.
You can reduce unnecessary prompt length, reuse shared prefixes where supported, place limits on unusually large requests, or separate latency-sensitive interactive traffic from heavy document-processing jobs.
You can also introduce additional serving capacity. But adding another GPU or server only helps when requests are distributed in a way that reduces contention.
A particularly useful distinction is between interactive requests and batch-oriented requests.
Interactive requests need predictable responsiveness. Batch-oriented tasks may tolerate waiting longer in exchange for efficient throughput.
When these workloads have very different requirements, isolating them can be more effective than forcing one scheduler configuration to satisfy both.
One GPU Does Not Have to Serve Every Workload the Same Way
A shared AI server can be organized around service objectives rather than treating every request as interchangeable.
Short conversational requests may receive tighter latency targets. Large document-processing jobs may be queued, rate-limited, or routed to separate capacity.
This does not eliminate the cost of computation. It makes that cost more predictable and prevents one class of workload from dominating the experience of another.
The Real Question Is Not Whether the GPU Is Busy
High GPU utilization is often presented as evidence of a healthy inference server.
It can be useful evidence, but it is incomplete.
A GPU can remain busy while users wait in a queue. It can remain busy while one long prefill delays other requests. It can remain busy while memory pressure forces the engine to reschedule work.
The goal of an interactive AI service is not simply to maximize the amount of computation performed.
It is to complete useful work while maintaining acceptable response times.
That requires measuring the experience of individual requests, especially when short and long workloads share the same hardware.
A Faster AI Server Is Sometimes a Better-Scheduled AI Server
Return to the employee who uploaded the large document.
The document still needs to be processed. Its tokens still require computation, and its context still occupies memory. Chunked prefill cannot make that work disappear.
What it can do is give the inference engine more flexibility in deciding how that work shares the GPU with other requests.
For a shared AI service, that flexibility can matter as much as raw benchmark speed.
The server users remember is not necessarily the one with the highest tokens-per-second result. It is the one that continues to feel responsive when everyone needs it at once.
