8GB vs 12GB vs 16GB VRAM for Local LLMs: What Models Actually Fit

Local LLM Hardware Guide

8GB vs 12GB vs 16GB VRAM: What Models Actually Fit?

The difference between an 8GB, 12GB and 16GB GPU is not just a few extra gigabytes. Each step changes which quantizations make sense, how much context you can keep, and whether a larger model runs entirely on the GPU or quietly spills into system RAM.

Choose my VRAM 8GB 12GB 16GB

Here is the trap with local LLMs: people look at the size of the GGUF file, compare it with the number printed on the GPU box, and assume the arithmetic is finished. A 7GB model file and an 8GB card? Perfect. An 11GB file and a 12GB card? Also perfect. Unfortunately, the GPU has other bills to pay.

The model weights are only the largest tenant. The KV cache grows as your conversation gets longer. The inference engine needs working buffers. The driver uses memory. Multimodal models may need extra components. And if you leave almost no headroom, a configuration that technically loads can still be unpleasant to use.

So this guide uses a stricter definition of fits: the model should fit at a useful quantization with enough room left for an ordinary context window and normal runtime overhead.

First, define “fit”

There are really three different meanings when someone says a local model “fits” on their GPU.

A quick way to estimate model weight size

Before downloading anything, parameter count gives you a rough first pass. A four-bit model theoretically needs half a byte per parameter. Real files are somewhat larger because quantization formats carry scales, metadata and tensors stored at other precisions.

As a rough planning rule, a good Q4-class GGUF often lands somewhere around 0.55–0.65GB per billion parameters. Treat that as a planning estimate, not a promise. The real file size is the number that matters.

That puts an 8B model roughly in the 4–5GB neighborhood, a 14B model around 8GB-ish, and a 24B model well into the low-to-mid teens depending on the quantization.

Now add the part people forget: the model has to run after it loads.

8GB

8GB VRAM: 8B is the comfortable lane

Plenty of useful local AI fits here — as long as you resist the urge to turn every download into a memory stunt.

Eight gigabytes is enough to run genuinely capable local models. The mistake is treating it as a miniature 16GB card. It is much better to use a good 7B or 8B model at a healthy quant than to force a much larger model into an ultra-aggressive quantization with almost no context headroom.

Comfortable 7B–8B Q4 / Q5 Good balance of quality, context space and speed.

Tight 12B Q4 Depends heavily on the exact model, quant and context.

Usually offload 14B+ Q4 Often too large for a comfortable full-GPU setup.

Qwen3-8B A natural 8GB target in a Q4 or Q5-class quant. Enough room remains for a practical local setup.

Llama 3.1 8B Another straightforward 8B-class workload where 8GB VRAM makes sense rather than merely survives.

This is also why an 8GB GPU can sometimes feel better with an 8B Q5 model than with a 14B model crushed into a very low-bit quant. More parameters do not automatically compensate for worse quantization, less context and slower CPU involvement.

12GB

12GB VRAM: this is where 14B becomes practical

Four extra gigabytes do more than the number suggests because they move an entire class of models out of “barely fits” territory.

Twelve gigabytes is a very useful local-LLM tier. Models around 12B to 14B can now be run at sensible Q4-class quantizations without immediately turning the rest of the VRAM budget into a crisis.

Qwen3-14B is a good example. The official model has 14.8 billion parameters. A Q4-class quant is therefore the kind of workload that naturally belongs much closer to a 12GB card than an 8GB one.

Comfortable 12B–14B Q4 The most interesting step up from the 8GB class.

Depends 14B Q5 / Q6 Exact quant size and context determine whether this remains pleasant.

Usually offload 20B–24B Q4 Weight files begin consuming almost the entire memory budget.

Qwen3-14B One of the clearest examples of why 12GB is meaningfully different from 8GB for local inference.

Gemma 3 12B A 12B multimodal model that belongs naturally in this VRAM class at practical low-bit quantization.

The catch is context. A model can have an advertised context window far larger than you should actually allocate on a 12GB card. Maximum context length and comfortable local context length are two completely different numbers.

16GB

16GB VRAM: more quality, or more model

This tier gives you a useful choice: run 14B models at better quantization, or start pushing into the 20B-plus range.

Sixteen gigabytes changes the conversation again. Instead of asking only “what is the largest model I can squeeze in?”, you can choose between a larger model and a higher-quality quant of a smaller one.

A 14B model at Q5 or Q6 can be a much nicer everyday setup than a borderline 24B model consuming almost every available gigabyte.

Comfortable 14B Q5 / Q6 Better precision while preserving useful context headroom.

Practical ~20B Q4 A sensible larger-model target when the exact file leaves headroom.

Tight 24B Q4 Possible with some builds, but often too close to the edge for large context.

Qwen3-14B at a higher quant A very comfortable use of 16GB: keep more precision and still have room for context.

Mistral Small 3.1 24B A useful example of the upper edge. Mistral itself positions the quantized 24B model for a 24GB RTX 4090-class GPU, so 16GB should be treated as aggressive-quant / limited-headroom territory rather than an effortless fit.

This is the point where people often discover that 16GB is not “almost 24GB.” A 24B or 27B-class model can be close enough to look tempting while still leaving too little room for the cache and runtime.

8GB vs 12GB vs 16GB: the practical table

Why context can ruin a perfect-looking VRAM calculation

Suppose your quantized model leaves 3GB free. That sounds generous until you ask for a long context window.

During inference, the model keeps key and value tensors for previous tokens. This is the KV cache. Longer prompts and longer conversations mean more cached state.

Qwen3-14B, for example, has 40 layers and eight KV heads. With a traditional 16-bit KV cache, a 32K-token sequence works out to roughly 5GB of cache before other allocations. Modern runtimes can use lower-precision KV cache formats, but the example shows why the advertised context maximum is not free.

This is why “128K context supported” should never be read as “128K context is sensible on my 8GB GPU.” Model architecture, KV precision and backend matter enormously.

More VRAM does not have to mean a bigger model

There is another way to spend those extra gigabytes: keep the same model and use a better quant.

Imagine you already like a particular 14B model. Moving from 12GB to 16GB does not force you to chase a 24B model. You can use Q5 or Q6 instead of Q4, keep a larger context window and still avoid CPU offloading.

For day-to-day use, that can be the better upgrade. A slightly smaller model that lives comfortably in VRAM is often more pleasant than a larger model constantly leaning on system memory.

Which VRAM tier should you target?

The simple answer

8GB: think 7B–8B. That is where the card can run a useful quant without constantly fighting the memory ceiling.

12GB: think 12B–14B. This is the point where models such as Qwen3-14B and Gemma 3 12B become much more natural local workloads.

16GB: think higher-quality 14B or roughly 20B-class Q4. Some 24B builds can be squeezed in, but the memory margin becomes thin quickly.

And regardless of the number on the GPU box, remember the rule that saves the most frustration: leave room for the model to run, not just to load.

FAQ