VRAM Calculation in Practice: Model Scale × Quantization, and How to Estimate KV Cache

The previous article covered "choose a card by VRAM." This one opens up the math: why does the same model need several extra gigabytes at different context lengths? When "27B won't run," is the gap in weights or in KV cache? How do you size GPU offload layers? Once the formulas and measurement methods are clear, you can estimate accurately before buying and tune precisely after deploying.

Context length is the multiplier nobody budgets

The single most common planning error is computing VRAM for short contexts and then running long documents. KV cache at 8K may be manageable while the same model at 32K doubles your total requirement — and users discover this only when the first long document arrives.

Before buying, write down the longest realistic input your workload will send: whole codebases, full documents, long conversation histories. Size the card for that number, not for the demo prompt. If the honest answer is "sometimes very long," either budget for it or plan a documented fallback (summarize-then-process, chunking) rather than discovering the failure in production.

A quick estimation workflow

Pulling it together, here is the workflow to run before any purchase or deployment decision:

1. Pick the target model and tier — for example, 27B at Q4_K_M.
2. Compute weights from the table above, then confirm against the model card's actual file size.
3. Estimate KV cache at your realistic context length and concurrency using the reference values, not your hopes.
4. Add overhead of 1-2GB, then target 85-90% of the card's capacity — never 100%.
5. Verify empirically once running: nvidia-smi for peaks, llama.cpp logs for KV cache, and a tokens-per-second benchmark at your real prompt length.

The gap between step 4 and step 5 is where most surprises live. Treating the formula as a first draft and the measurement as the truth is the whole discipline.

Formula one: weight size

Weight size = parameter count × bytes per parameter. Empirical coefficients per tier:

Bandwidth matters as much as capacity

Two cards can hold the same model and deliver very different speeds. Generation speed (tokens per second) is dominated by memory bandwidth: the model weights are read for every token generated, so bandwidth sets the ceiling.

This changes purchase logic: a card with 24GB of slower memory often beats a card with 16GB of faster memory for local inference, because the constraint is usually capacity, not speed — while the opposite is true once capacity is satisfied and speed becomes the complaint.

A practical benchmark habit: before buying, run the exact model and tier you plan to use on a friend's machine, a rental, or a demo, and note tokens per second at your realistic context length. Spec sheets do not predict inference speed as reliably as one honest measurement.

Formula two: KV cache

KV cache stores the K/V matrices of historical tokens and is the hidden big-ticket item in long-context deployment. Per-request estimate:

KV cache ≈ 2 (K and V) × layers × total KV head dimension × context length × 2 bytes (fp16)

You do not need the exact formula — just two rules:

1. Grows linearly with context — 8K to 32K means roughly 4x the KV cache
2. Grows linearly with batch — 4 concurrent requests means roughly 4x the KV cache

Practical reference values (fp16, single request, for estimation only):

Formula three: runtime overhead

Framework overhead: CUDA context (~0.3-0.5GB), inference engine caches, memory fragmentation — budget another 1-2GB. With multiple GPUs, each card carries its own overhead.

Full worked examples

Case 1: Qwen3.8-27B, Q4_K_M, 8K context, single user

• Weights 16GB + KV cache ~1.5GB + overhead 2GB ≈ 19.5GB → a 24GB card runs it

Case 2: same, but 32K context

• Weights 16GB + KV cache ~6GB + overhead 2GB ≈ 24GB → a 24GB card sits right at the edge; 32GB or a lower tier is safer

Case 3: Hunyuan 4B (hy4), Q4_K_M, 8K context

• Weights ~2.5GB + KV cache ~0.5GB + overhead 2GB ≈ 5GB → runs easily on 8GB, acceptable even on CPU

Sizing GPU offload layers

llama.cpp's -ngl (n_gpu_layers) decides how many layers live in VRAM versus RAM. Method:

1. Try all layers: set -ngl to the total layer count (the startup log prints it) and see if it OOMs
2. Drop on OOM: reduce by 4-8 layers at a time until it loads
3. Weigh the speed: more layers left in RAM means slower generation — benchmark tokens/s with the same prompt to find the "loads plus acceptable speed" balance

Calibrate by measurement — more accurate than formulas

Formulas estimate before buying; measurement tunes after deploying:

Watch VRAM usage: run nvidia-smi in another terminal and observe peak usage during inference.

Read the KV cache allocation: llama.cpp prints the actual KV cache allocation at startup (something like llama_kv_cache: ... MiB) — use it to calibrate your coefficients.

Check fragmentation: if peak usage runs 2GB+ above the formula, check whether you set -c (context) or batch parameters larger than needed.

# Watch the KV cache allocation in the logs when starting llama-server
./llama-server \
  -m ./models/Qwen3.8-27B-Q4_K_M.gguf \
  -ngl 40 \
  -c 8192 \
  --host 127.0.0.1 --port 8080

Batched inference multiplies VRAM

For multi-user local deployment: KV cache multiplies by concurrency. One user at 8K context uses ~1.5GB; four concurrent users use ~6GB — plus weights and overhead, a 24GB card running 27B with 4-way concurrency gets tight.

Strategy: cap concurrency (llama-server's parallel setting), reserve VRAM for it, or hand high-concurrency scenarios to a cloud API.

FAQ

Related reading

• Buying Local Compute: A GPU Purchase Guide — choosing a card by VRAM
• The Complete Local Deployment Playbook — hardware check to workflow integration
• Qwen KV Cache and Long Contexts — KV cache mechanics
• Qwen GGUF Local Deployment Primer — deployment basics
• Local Compute Cost Model — buy vs rent vs API