Buying Local Compute: A GPU Purchase Guide for Running Open Chinese Models
Buying Local Compute: A GPU Purchase Guide for Running Open Chinese Models
If you want to buy your own hardware to run open Chinese models like Qwen, DeepSeek, and GLM locally, the first question is almost always "which GPU." And there is a common misconception here: staring at GPU model names and benchmark charts. For local inference, what actually decides "can it run and at which tier" is VRAM capacity — the model number decides speed, the capacity decides possibility. This guide gives you an executable purchase logic: compute VRAM first, then pick the card, then build the machine.
The short version
VRAM decides how large a model you can run: 8GB handles 7-8B quantized, 16GB handles 14B or 27B at low quantization, 24GB handles 27B at mainstream tiers, and 48GB+ is where 70B-class models get comfortable. Start by estimating weight size from model scale and quantization tier, add KV cache and runtime overhead to get your target VRAM, then shop for a card — within the same capacity, choose by budget and power draw.
Step 1: compute your target VRAM
Before picking a card, answer one question: how large a model, at which quantization tier? Those two parameters decide weight size.
Quick estimation: parameter count × bytes per parameter. Rough byte coefficients per tier (they differ slightly across models; use the actual GGUF file sizes on the model card for precision):
| Tier | Bytes/param (approx.) | 27B weights (approx.) |
|---|---|---|
| Q8_0 | 1.05 | 28GB |
| Q6_K | 0.85 | 23GB |
| Q5_K_M | 0.72 | 19GB |
| Q4_K_M | 0.60 | 16GB |
| Q3_K_M | 0.48 | 13GB |
| Q2_K | 0.35 | 9.5GB |
Why Q4_K_M is the baseline
Q4_K_M is the community's most-used tier: about one third the size of FP16 with quality loss acceptable on most tasks. Use it as the "main tier" when estimating VRAM, leave headroom upward (Q5/Q6/Q8), and treat lower tiers (Q3/Q2) as fallbacks when VRAM is tight.
Then add two more costs:
KV cache — grows linearly with context length and is the hidden big-ticket item in long-context deployment. The exact formula and measurement methods are in VRAM calculation in practice.
Runtime overhead — CUDA context, framework caches, fragmentation; usually another 1-2GB.
Example: running Qwen3.8-27B at Q4_K_M (about 16GB of weights) plus 8K-context KV cache (roughly 1-2GB) plus 2GB overhead ≈ 19-20GB — a 24GB card runs it comfortably.
Step 2: match your VRAM tier
Below is a "VRAM capacity → typically runnable models" mapping (models illustrative of the open Chinese mainlines; verify against each model's actual GGUF file sizes):
| VRAM | Typically runs | Notes |
|---|---|---|
| 8GB | 7-8B class at Q4; 0.3-4B at any tier | Entry experiments, edge models, classification/extraction |
| 12GB | 8-14B at Q4; 27B at heavy compression | Light creation, coding assistance |
| 16GB | 14B at Q4/Q5; 27B at Q3 or partial offload | A value sweet spot for light-to-medium tasks |
| 24GB | 27B at Q4/Q5 comfortably; 32B class at Q4 | The mainstream pick for consumer local inference |
| 32-48GB | 27B at Q8; 70B class at Q4 with partial offload; MoE models with room | Quality-first, or larger open models |
| 80GB+ / multi-GPU | 70B-100B class; a feasible starting point for flagship MoE | Private deployment, team sharing |
MoE models store weights by TOTAL parameters
For MoE models like Qwen-AgentWorld-35B-A3B, only about 3B parameters activate per token, so inference speed and compute needs resemble a 3B model — but the full weights still take the size of 35B. VRAM must be planned by total parameters, not active. This is the most common misconception in local MoE deployment.
Step 3: choosing among cards with the same VRAM
Once capacity is fixed, same-capacity cards differ mainly in three ways:
Memory bandwidth. Generation speed (tokens/s) depends heavily on memory bandwidth — a card with double the bandwidth can be nearly twice as fast on the same model. On a budget, large capacity with moderate bandwidth often beats small capacity with high bandwidth.
Power draw and cooling. High-power cards need matching PSUs and case airflow; for always-on operation, electricity and noise are real costs.
Used and ex-mining cards. The second-hand market has value, but cards whose memory ran hot for long periods carry higher failure risk. Local inference is sensitive to memory stability (long contexts keep memory pressure high), so prioritize usage environment and warranty over price alone.
Not just the GPU — budget these too
- System RAM: for CPU offload or hybrid inference, RAM must hold the offloaded weights — 32GB minimum, 64GB recommended for 27B and above
- PSU: leave 30%+ headroom over peak system draw
- Case and airflow: long card dimensions and multi-card spacing are often overlooked
- Electricity: for always-on setups this is an ongoing cost, not a one-time one
Step 4: special cases
macOS unified memory. Apple Silicon's unified memory can be used directly by the GPU, and community support for running quantized models is mature (mlx ecosystem, Ollama builds). Good if you already use a Mac and want local models on the side; the tradeoff is usually slower generation than a same-price discrete GPU, and no upgrade path.
Multi-GPU. llama.cpp splits models across cards by layer and needs no specialized interconnect. When the model exceeds a single card (70B class, flagship MoE), two mid-tier cards with large combined capacity may beat one flagship card — at the cost of deployment complexity and power.
Pure CPU. Runs with enough RAM, but generation speed is typically tens of times slower than GPU. Only for validating a pipeline or for very small models (0.3-4B class), not for daily use.
Step 5: three cost models before you buy
Buying is not the only option. Split with one question: what is your usage pattern?
| Model | Fits | Characteristics |
|---|---|---|
| Buy (local compute) | High volume, long term, sensitive data | One-time spend plus electricity; marginal cost trends to zero; data never leaves the machine |
| Rent GPUs | Short experiments, peak supplement | Hourly billing; no ops; good for validation and temporary scale |
| Cloud API | Low volume, spiky, flagship quality | Per-token billing; zero ops; high capability ceiling |
The rule of thumb: local purchase starts to make sense when your monthly bill at cloud-API rates exceeds the monthly amortization of the target card. For the full three-way cost model and break-even points, see the local compute cost model. If you just want to validate "what local Chinese models feel like," starting with a cloud API is the cheapest way to try.
FAQ
What is the cheapest entry setup for running open Chinese models?
Does VRAM matter more than compute?
Can a 27B model run on a 16GB card?
Are MoE models lighter on VRAM?
Is buying a used GPU worth it?
How do I include electricity in the cost?
What GPU prices should I reference?
Related reading
- VRAM Calculation in Practice — the math behind this guide
- The Complete Local Deployment Playbook — from hardware check to workflow integration
- Local Compute Cost Model: Buy, Rent, or API — the three-way cost comparison
- Qwen GGUF Local Deployment Primer — deployment basics
- GGUF Quantizer Comparison — how to choose quantization tiers