The model loads. You still have free VRAM. Short prompts work perfectly. Then you paste a long document or raise the context limit — and the same setup suddenly runs out of memory. The missing piece is the memory that grows with the conversation.
Loading a local LLM is only the first memory test. The model weights may fit comfortably in your GPU, but inference creates another large allocation: the KV cache. Unlike the weights, that cache grows as more tokens remain in context.
This is why a model can be perfectly stable at 4K or 8K context and fail at 32K without the model file changing by a single byte.
The model is only part of the VRAM bill
Imagine a 12GB GPU and a quantized model that occupies 8.5GB once loaded. At first glance, you have 3.5GB left. That sounds comfortable.
But the GPU also needs working buffers, backend allocations and memory for the active context. The largest growing component is usually the KV cache.
The cache stores key and value tensors from previous tokens so the model does not have to rebuild the entire conversation every time it predicts the next token.
Model-weight memory is mostly fixed. Context memory grows. A configuration with plenty of space at 4K can cross the VRAM limit at 16K, 32K or 40K.
A real example: Qwen3-14B
Qwen3-14B has 40 transformer layers, eight key/value heads and a head dimension of 128. With a traditional FP16 or BF16 KV cache, one retained token needs about 160 KiB of key/value storage.
One token is tiny. Thirty-two thousand tokens are not.
| Context | Approx. FP16 / BF16 KV cache | What it means |
|---|---|---|
| 4K | ≈ 0.63 GiB | Usually easy to absorb if the model leaves reasonable VRAM headroom. |
| 8K | ≈ 1.25 GiB | The cache has doubled while the model itself is exactly the same. |
| 16K | ≈ 2.50 GiB | A previously comfortable GPU can now begin to feel tight. |
| 32K | ≈ 5.00 GiB | Context memory alone has become a major part of the VRAM budget. |
| 40K | ≈ 6.25 GiB | Long context can exceed the free space left after model loading. |
Why the failure feels sudden
Nothing looks wrong at first. You can chat with the model, write code and run short prompts without trouble. Then one long PDF or a conversation that has been growing for an hour pushes the cache past the remaining VRAM.
The last prompt gets blamed because that is when the error appears. But the real cause is usually the amount of history that has accumulated before it.
On a shared inference server, the problem can arrive even sooner because several sequences may need KV memory at the same time.
What actually fixes it?
If you normally need 8K or 16K, allocating 32K by default simply spends memory on capacity you are not using.
Q8 or Q4 KV formats can cut the context-memory cost substantially. Backend support and output quality still deserve testing.
A slightly smaller quantized model that leaves 2–3GB free can be more useful than a larger model that consumes almost the entire card.
For servers, context is multiplied across concurrent work. Fewer active sequences can dramatically reduce total memory pressure.
llama.cpp gives you direct control
You do not have to accept the largest context a model supports. Set the working window you actually need, and choose the cache format separately from the model-weight quantization.
A model that supports 40K context does not need to reserve a 40K working window on every machine. Choose the context that matches the task and the memory you actually have.
What to check before choosing a model
| Check | Why it matters |
|---|---|
| Quantized weight size | Shows roughly how much of your VRAM budget is already committed before generation begins. |
| KV-head count | Fewer KV heads can dramatically reduce cache memory at the same context length. |
| Context target | 8K and 32K are very different memory workloads even when the model is identical. |
| KV-cache precision | FP16, Q8 and Q4 can change the context-memory requirement substantially. |
| Free VRAM after load | If almost nothing remains after loading the model, long context is already going to be difficult. |
The simple explanation
A local LLM can fit perfectly in VRAM and still fail later because the conversation has its own memory cost.
The model weights stay approximately fixed. The KV cache grows with the active context. Double the retained tokens and, for the same architecture and cache format, you roughly double the KV-cache memory.
So if a model works at 8K and fails at 32K, the model did not suddenly become larger. The context did.
FAQ
Why does longer context use more VRAM?
The model retains key and value tensors for previous tokens. More tokens in the active context require a larger KV cache.
Does 32K context use four times as much KV memory as 8K?
For the same model, cache format and sequence configuration, KV-cache memory scales approximately linearly with the number of retained tokens.
Does Q4 model quantization automatically mean a Q4 KV cache?
No. Model-weight quantization and KV-cache precision are separate settings in runtimes such as llama.cpp.
Is KV-cache quantization always worth using?
It can save a large amount of memory, especially at long context, but performance and output quality should be tested with the model and backend you actually use.
Is lowering context better than CPU offloading?
If you do not need the longer context, lowering the context window is usually the cleaner solution because it removes the memory demand rather than relocating part of the workload.
