The AI Model Is Fast. The App Still Makes You Wait.

Conceptual machine with a spinning metal flywheel behind a narrow gate and a long queue of paper envelopes, illustrating a fast model inside a slower process. Inference
A model can finish quickly while the app keeps you waiting. Follow the request through retrieval, streaming, proxies and rendering to find which delay actually reaches the user.

The model finishes its answer in three seconds. The application takes eight seconds to show it. Both numbers can be correct.

A model request sits inside a larger operation: the app may authenticate the user, fetch documents, wait for inference capacity, call another service, validate the result and send it through a proxy before the browser displays anything. A fast generation engine shortens only the portions of that path it actually controls.

To understand AI app latency, follow the user’s request from the click to the first useful answer on screen. That boundary is wider than a model benchmark.

One answer has several clocks

“Time to first token” needs a start point. An inference server might measure from receipt of its request; an SDK might measure from the client’s API call. Neither includes document retrieval completed before that call. A browser measurement starts earlier still.

Likewise, tokens per second describes how output arrives during a chosen interval. It does not tell you how long the application spent assembling the request, or how long it held the answer afterwards. These are the different timing boundaries behind inference benchmarks.

Pick two user-facing milestones: the first useful content becomes visible, and the requested result becomes usable. They need not coincide. A paragraph can stream before the answer is complete; a structured action might require a fully parsed, validated response.

Doubling generation speed does not halve the wait

Consider an illustrative application that waits for a complete response before displaying it. Assume its stages are serial and generation takes three of the total eight seconds. Halve generation time while leaving everything else unchanged, and the whole operation falls to 6.5 seconds—a reduction of 18.75%, not 50%.

Where eight seconds went

An assumed, fully buffered request. Every row follows the previous one; the bars compare durations rather than positions on a shared timeline.

App checks0.4 s
Retrieve context2.6 s
Queue + prefill0.8 s
Generate output3.0 s
Validate + deliver1.2 s
8.0 sOriginal total
6.5 sWith generation twice as fast
Original calculation using assumed timings, not measurements from a provider or IL.DIGITAL. Total: 0.4 + 2.6 + 0.8 + 3.0 + 1.2 = 8.0 s. Faster case: the 3.0 s stage becomes 1.5 s. Real stages may overlap; do not add overlapping spans as if they were serial. The one-time emphasis is not an execution replay.

This arithmetic also explains why a smaller model can disappoint. If most delay sits in a document service, changing the generator leaves that dependency untouched. Conversely, reducing a slow retrieval stage may improve the experience without changing model hardware at all.

The first streamed event might contain no answer

Streaming lets the client receive pieces before the entire response completes. That creates an opportunity for earlier display, not a guarantee. The application still has to forward, interpret and render those pieces.

Anthropic’s streaming documentation makes the distinction concrete. Its event stream includes message-start events, content-block events and pings; content can also include tool-input or thinking blocks. The first received event is therefore not necessarily the first text a user can use.

A tool-using workflow adds another boundary. The model may first request a lookup, wait for its result and only then produce the answer. Fast initial output might simply mean it asked for a slow external operation quickly.

For a text assistant, record when the browser first renders useful answer text. For a workflow that must approve a transaction or produce valid JSON, measure readiness at the point where those required checks finish. A spinner, heartbeat or incomplete data object should not quietly become the success metric.

The network can deliver in batches even when the model streams

A reverse proxy sits between the application and the user. Depending on its configuration, it can buffer an upstream response rather than pass each arriving piece through immediately.

The NGINX proxy module documentation describes both behaviours. Buffering is enabled by default; with buffering disabled, the response is passed to the client as it arrives. The response header X-Accel-Buffering can also influence this, unless the proxy is configured to ignore it.

This is one possible source of bursty display, not a diagnosis of every pause. The application might itself collect a whole answer, validate a structure or update the screen infrequently. Check the intervals at each boundary: server emits, client receives, interface renders. A healthy connection alone cannot establish that the last step happened promptly.

Trace one request instead of subtracting unrelated averages

A distributed trace ties the work for one request together. OpenTelemetry describes traces as related spans, with each span representing an operation and carrying timing and context. That allows an app’s retrieval, model calls and external requests to be examined as parts of one journey.

The important delay is along the chain of dependencies that determines completion. If two independent lookups overlap, adding their durations exaggerates the wait. If the next model call needs both results, the slower lookup determines when that step can begin. Parallelising operations helps only where their inputs and correctness requirements allow it.

Measure elapsed time within each component using an appropriate monotonic clock. The browser Performance API provides timing designed for duration measurements. Subtracting raw timestamps from unrelated machines can introduce clock-offset error; correlating a request does not synchronise every clock.

Inspect slow requests as well as typical ones. An acceptable median can coexist with a poor 95th percentile, where roughly the slowest five percent of observations begin. Keep comparisons aligned by workload, concurrency, output length and cache state.

Improve the part the user is actually waiting for

If prompt processing dominates, reusing a compatible prompt prefix may help. If a retrieval service dominates, its query or data path deserves attention. If useful text arrives promptly but appears late, inspect delivery and rendering. These are different interventions for different evidence.

Retries deserve their own trace entries. A timeout may hide a second attempt and an additional model call; it can also create uncertainty about an external action that already succeeded. Shorter timeouts are not automatically a faster or safer experience.

After a change, compare the same user-facing milestone under comparable traffic and confirm that answer quality and correctness survived. The meaningful improvement is the time removed between the user asking and the result becoming useful—not a faster number measured somewhere inside that interval.

Sources

Rate article