- Key Takeaways
- On this page
- The browser owns a connection. The engine owns the work.
- Define what should survive
- A proxy can change what the application sees
- A disconnect notification needs a cleanup path
- 5 checks for LLM request cancellation
- Unwanted tokens still occupy a real machine
- Measure energy rather than assigning a price to every token
- Why VRAM can stay allocated after cancellation
- Look for recovered serving capacity
- How to measure LLM request cancellation
- Detection delay
- Abort completion
- Post-cancel work
- Remaining users
- A timeout needs its own policy
- il.digital Lab: test the complete cancellation path
- Start with one request that should no longer exist
LLM request cancellation should stop work that nobody needs anymore. Yet you can ask your local AI server for a long report, read the first few lines and close the tab while the GPU keeps generating the rest.
The model can continue producing tokens, retaining conversation state and competing with other requests. Nothing has crashed. The machine is still doing the work it was asked to do because the message that you no longer want it has not reached the component performing the computation.
Closing a chat can stop generation, but only when cancellation travels through the serving stack. The browser, reverse proxy, application and inference engine each have a separate role. Following that path reveals whether Stop actually releases capacity for somebody else.
Request ownership
The browser owns a connection. The engine owns the work.
A chat interface can stop appending text while its network request remains open. It can also abort that request. These actions look similar on screen but create different signals upstream.
The browser’s AbortController API can abort a fetch operation and response-body consumption. LLM request cancellation still needs server-side handling to connect that interruption to the correct generation.
Consider an application that starts a background worker and streams its output. Ending the stream may leave the worker alive. That can be intentional for a saved report, but wasteful for an abandoned chat answer with no remaining consumer.
- 1. Client stops The interface aborts its request or sends an explicit cancellation.
- 2. Proxy responds The intermediary handles the disconnect according to its configuration.
- 3. Application cleans up The handler closes upstream work or forwards an identified cancellation.
- 4. Engine aborts Generation ends and request-owned serving capacity becomes reusable.
A proxy can change what the application sees
A reverse proxy creates another connection boundary. Your application may be connected to the proxy rather than directly to the person who closed the tab.
NGINX exposes relevant behavior through
proxy_ignore_client_abort.
Its documented default is off; the directive
controls whether the upstream connection should close when
the client leaves before receiving the response.
Response buffering is separate.
proxy_buffering controls how NGINX receives and
stores upstream response data. Disabling it can help streaming
delivery, but does not establish an application-to-engine
abort contract. See the
NGINX proxy module reference
.
For LLM request cancellation, inspect the effective configuration on the deployed path. An intermediary that preserves upstream work changes where, and when, the departure becomes observable.
Application boundary
A disconnect notification needs a cleanup path
The
ASGI HTTP specification
defines an http.disconnect receive event and
describes exceptions when sending on a closed connection,
with implementation and version caveats.
These signals provide an opportunity for cleanup. They do not automatically identify every background task that the application created.
Reliable LLM request cancellation connects the browser-facing request to its upstream generation identity. Cleanup then has a target: close the upstream stream, cancel the associated task or invoke the supported engine abort mechanism.
In vLLM’s documented AsyncLLM generation path , cancellation or generator closure is caught and the associated request is aborted. That demonstrates an engine-side mechanism. A custom gateway must still reach it correctly.
Verify that integration on the deployed version. An available abort method cannot stop a generation whose identity the application has lost.
Diagnosis
5 checks for LLM request cancellation
Follow the evidence across each boundary. A visible change in the interface is only the first observation.
| Observed behavior | Possible explanation | Evidence to inspect |
|---|---|---|
| 1. Text stops appearing | Rendering stopped while the request continued. | Client abort event and network-request lifetime. |
| 2. Browser request ends; application work continues | An intermediary or detached task preserved upstream work. | Proxy logs, handler exit and upstream connection closure. |
| 3. Handler exits; generation remains active | Cleanup did not reach the correct engine request. | Request identity, abort invocation and terminal state. |
| 4. Generation ends; VRAM stays allocated | The engine retained its memory pool for reuse. | Active requests and reusable KV blocks. |
| 5. GPU remains busy | Other requests are using recovered capacity. | Per-request traces and remaining users’ latency. |
Physical layer
Unwanted tokens still occupy a real machine
A generation that remains active continues participating in the workload. Its attention state occupies KV-cache capacity, and further tokens require accelerator work.
On a shared server, failed LLM request cancellation can affect people who are still waiting. Our explanation of continuous batching shows how active requests compete for the GPU.
Removing unwanted work can create room for useful requests. The improvement depends on cache pressure, execution time and queueing. There is no universal performance gain.
Why VRAM can stay allocated after cancellation
Successful LLM request cancellation does not unload the model. Serving engines can also retain allocated memory for cache pools.
vLLM’s KV-cache design distinguishes request-associated blocks, reference counts and a free-block queue. Blocks become reusable when active references are released. Shared blocks can remain in use, and cached content may survive until eviction.
Capacity available to another request is different from memory returned to the GPU driver. A large process-level allocation alone does not prove a leak.
Measurement
How to measure LLM request cancellation
Record the client cancellation, application observation, engine abort and terminal request state. These timestamps separate detection delay from cleanup delay.
Count tokens generated after cancellation at the engine. Do not substitute chunks arriving late in the browser: buffered output can arrive after computation has stopped.
Trace LLM request cancellation using a correlation ID across the gateway and engine. Prompt contents are unnecessary for identifying request lifetimes.
Detection delay
Client cancellation to application observation, accounting for clock alignment.
Abort completion
Application observation to the engine’s terminal request state.
Post-cancel work
Tokens generated after cancellation, attributed to the request.
Remaining users
Queue time, time to first token and streaming gaps.
The vLLM metrics documentation lists running requests, generated tokens, KV-cache usage and latency. Aggregate counters describe the server, but cannot identify which request produced a token. Pair them with request-level traces.
Test before the first token, too. A cleanup check that only runs when output is yielded may behave differently during queueing or prompt processing.
A timeout needs its own policy
A timeout tells one component to stop waiting. Whether it ends upstream computation depends on the cleanup path.
If a retry starts while the original generation remains active, both attempts can compete for resources. This resembles the uncertainty behind agents repeating actions after a timeout , although duplicated inference and duplicated business actions have different consequences.
Combine LLM request cancellation with an overall deadline and sensible output limits. Forward the remaining time budget where supported. A silent network failure may be detected later than an explicit Stop action.
For agents, ending generation also does not reverse a tool action that already happened. External operations need their own execution and recovery rules.
Start with one request that should no longer exist
Send a long generation through the actual serving path. Cancel it, locate its identity in application and engine records, and establish when it stopped receiving work.
If it continues, inspect each boundary. If memory stays allocated, check whether capacity is reusable. If the GPU remains busy, identify the requests now using it.
Successful LLM request cancellation means an unwanted answer no longer competes with someone who is still waiting. A disappearing paragraph proves that the interface stopped displaying text. The request lifecycle tells you whether the machine stopped doing the work.
