Conceptual machine with a spinning metal flywheel behind a narrow gate and a long queue of paper envelopes, illustrating a fast model inside a slower process.Inference
The AI Model Is Fast. The App Still Makes You Wait.
054
A model can finish quickly while the app keeps you waiting. Follow the request through retrieval, streaming, proxies and rendering to find which delay
Conceptual illustration of a busy GPU sharing its processing capacity across several AI response streams.Inference
A GPU Can Be 100% Busy and Still Deliver Slow AI Answers
0153
A GPU-utilization meter counts activity, while you experience waiting time. Understanding the gap explains why a fully occupied accelerator can still deliver
A local LLM fitting inside GPU VRAM while translucent context-memory layers expand and fill the remaining space.Inference
Why a Model Fits in VRAM but Still Runs Out of Memory at Long Context
0325
Your local LLM fits in VRAM — until you increase the context window. Here is why KV cache can consume several extra gigabytes and trigger an OOM.
A 30B local LLM overflowing from a 12GB GPU memory chamber into system RAM, illustrating partial GPU offloading.Hardware
Can You Run a 30B LLM on 12GB VRAM? What Actually Happens
0335
A 12GB GPU can run a 30B local LLM — but usually not entirely in VRAM. Here is what quantization, CPU offloading and context memory actually do.
Inference
LLM Prefix Caching Explained: Why Repeated Prompts Can Start Faster
0199
Two requests can contain almost the same 20,000-token prompt and still reach the first generated token at very different times. On the first request, the
Inference
How Long Prompts Disrupt Shared LLM Inference
0243
Three people are using the same local AI server. Two answers are already streaming at a comfortable pace. Then a third user pastes a long document and presses Send.
Inference
When Does a Second GPU Actually Help LLM Inference?
0212
There is an appealingly simple upgrade path for a busy local AI server: install another GPU. Twice the accelerators should mean something close to twice
Inference
How Many Users Can One GPU Actually Serve?
1311
A local AI server can feel almost instantaneous when one person is using it. Add a second conversation and it may still feel fine. Add several people with
Inference
Your LLM Benchmark May Be Measuring the Wrong Thing
1246
A local LLM produces 100 tokens per second. On paper, that sounds fast. Then a real user sends a long prompt and waits several seconds before anything
Inference
Prompt Processing vs Token Generation: Why Local LLM Speed Has Two Numbers
4635
A local LLM does not have one meaningful “tokens per second” number. Before it can write the first token of an answer, it has to process the tokens already in the prompt.