hy4 GGUF Local Deployment in Practice: Quantization Tiers, VRAM Math, and llama.cpp / Ollama Commands
hy4 GGUF Local Deployment in Practice: Quantization Tiers, VRAM Math, and llama.cpp / Ollama Commands
hy4 is the most self-hostable model in the Hunyuan family: a 4B dense model with official FP8, GPTQ-Int4 and AWQ-Int4 weights, plus more than half a dozen community authors publishing GGUF conversions. What actually complicates the job is not the model but three decisions you have to make yourself: which quantization tier, how much memory to reserve, and which runtime to use. This walkthrough follows the order that works in practice — decide, measure, deploy, troubleshoot — and every command can be copied as-is.
The 30-second version
The recommended first run for hy4 is a community Q4_K_M GGUF served with llama.cpp (llama-server for an OpenAI-compatible endpoint). Budget roughly 2.5GB for weights, 0.5GB for the KV cache, and 0.5GB of runtime overhead, which fits comfortably in 6GB. On a Mac, swap the GGUF for the mlx-community 4bit conversion.
Step one: pick the quantization tier
| Tier | Measured file size (bartowski repository) | Quality | Fits |
|---|---|---|---|
| IQ4_XS | 2.37 GB | Close to Q4 | An equivalent Q4 route when memory is tight |
| Q3_K_M | 2.17 GB | Perceptible loss | Barely workable on a 4GB card |
| Q4_K_M | 2.61 GB | Nearly lossless | Default choice, enough for most work |
| Q5_K_M | 3.03 GB | Close to original | Take it when memory allows |
| Q6_K | 3.47 GB | Almost identical to original | A conservative pick for long generations |
| Q8_0 | 4.49 GB | Essentially original | Accuracy baseline comparisons |
| FP16 (original) | ~8.4 GB | Original | Only for an FP16 inference stack |
The original weight size follows the model card parameter count: Hunyuan-4B-Instruct has 4.22 billion parameters, which is about 8.4GB in BF16.
The practical order of choosing
Start with Q4_K_M and confirm the command line, context length, and serving all work. If your task is precision-sensitive, such as structured extraction or code generation, compare outputs against Q5_K_M. Only build an accuracy baseline, which means measuring how much quantization costs you, when you need Q8_0 or FP16. Starting at FP16 usually just wastes memory.
Community publishers of hy4 GGUF:
| Publisher | Repository character |
|---|---|
| bartowski | Widest tier coverage and the highest download count, a good default source |
| mradermacher | Publishes both standard quants and an i1-series variant |
| gabriellarson | Mostly standard quants |
| DevQuasar | Mostly standard quants |
| Disya | Primarily Q8_0, useful for a high-precision comparison |
| Edge-Quant | Single-tier Q4_K_M for a minimal choice |
Read the quantization marker in the filename (Q4_K_M, Q5_K_M, Q8_0) and the source repository, and do not treat same-named files from different authors as the same artifact.
Step two: compute memory, and do not stop at the weights
Total memory is three parts added together:
total memory ~= weight footprint + KV cache + runtime overhead
Weight footprint: hy4 has 4.22 billion parameters multiplied by bytes per parameter. Q4_K_M is about 0.65 bytes per parameter, Q8_0 about 1.06, and FP16 is 2.
KV cache: it grows linearly with context length. hy4's official configuration is 36 layers, 8 KV heads, head dimension 128, which works out to roughly 144KB per token in BF16:
KV cache ~= 2 x layers x KV heads x head dim x context length x bytes
~= 2 x 36 x 8 x 128 x context length x 2 bytes
~= 144KB x context length
For reference: a 4K context costs about 0.56GB and a 32K context about 4.6GB. The official maximum position length is 262144 (256K), but filling that window means the KV cache alone eats more than ten GB, so the context you can actually afford has to be derived backwards from your memory, not copied from the label.
hy4 is a dense 4B model, so the KV cache stays moderate, and that is exactly why it is friendlier to long-context work than a same-size MoE: an MoE activates only part of its parameters per token, but all total weights still occupy memory.
Runtime overhead: the CUDA context, compute buffers, and quantization metadata, typically 0.3 to 1GB.
A quick lookup
Q4_K_M (2.61GB) with a 4K context: about 2.6 + 0.6 + 0.5 = 3.7GB, tight but workable on a 4GB card. Q4_K_M with a 32K context: about 2.6 + 4.6 + 0.5 = 7.7GB, which needs 8GB or more. Q5_K_M with an 8K context: about 3.0 + 1.1 + 0.5 = 4.6GB, fine on a 6GB card. Q8_0 at 4.49GB already eats most of a card. These are estimates; the real number is what nvidia-smi reports right after the first load.
Step three: run it with llama.cpp
# 1. Fetch a community GGUF (bartowski shown; adjust the filename to the repo listing)
huggingface-cli download bartowski/tencent_Hunyuan-4B-Instruct-GGUF \
Q4_K_M.gguf --local-dir ./hunyuan4b
# 2. Talk to it directly first, to confirm the weights load
./llama-cli -m ./hunyuan4b/Q4_K_M.gguf -p "用三句话解释什么是 MoE" -n 256
# 3. Expose an OpenAI-compatible server
./llama-server -m ./hunyuan4b/Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 \
--ctx-size 32768 \
--jinja
With the server up, an existing client changes two parameters:
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:8080/v1",
api_key="not-needed",
)
resp = client.chat.completions.create(
model="local-model",
messages=[{"role": "user", "content": "你好，介绍一下你自己"}],
)
print(resp.choices[0].message.content)
The model name can be anything, but do not use a real API ID
llama-server does not strictly validate the model name, so a local service can be called with local-model. If your client looks the model name up against a remote catalog, keep the local and cloud endpoints clearly separated.
Step four: run it with Ollama for the best tool compatibility
Ollama ships its own model management and API ecosystem, and tools such as Open WebUI, Dify, and Cursor understand it out of the box.
# 1. Create the model from a local GGUF through a Modelfile
cat > Modelfile <<'EOF'
FROM ./hunyuan4b/Q4_K_M.gguf
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
PARAMETER num_ctx 8192
PARAMETER temperature 0.7
EOF
ollama create hy4-local -f Modelfile
# 2. Verify
ollama run hy4-local "写一个 Python 函数解析 ISO 8601 时间"
# 3. Inspect and clean up
ollama list
ollama show hy4-local
ollama rm hy4-local
Always take the template from the model card
A mismatched chat template is the most devious failure mode: the model loads, it produces tokens, but the answers are bad, so it looks like the model itself is weak. Copy the template from the Usage section of the matching model card. Hunyuan templates differ across generations, so copy carefully.
Step five: the Mac and edge routes
| Platform | Route | Notes |
|---|---|---|
| Apple Silicon | mlx-community/Hunyuan-4B-Instruct-4bit | Load directly with mlx-lm; unified memory makes quantization pay off immediately |
| Edge and embedded | taobao-mnn/Hunyuan-4B-Instruct-MNN | Alibaba's MNN inference engine, aimed at mobile and embedded |
| Older CPUs | Community Q4/Q5 GGUF | The llama.cpp CPU backend, where 4B is the realistic ceiling |
# Quick MLX verification
pip install mlx-lm
python -m mlx_lm.generate --model mlx-community/Hunyuan-4B-Instruct-4bit \
--prompt "用一句话解释向量数据库" --max-tokens 128
How to get a trustworthy speed estimate
Do not trust someone else's tokens-per-second number. CPU, memory bandwidth, context length, and backend implementation all change the result. The right method is to measure on your own machine: fixed prompt, fixed generation length, three runs, take the median, and record the peak memory at the same time. Keep that number as your baseline so a later quantization change stays comparable.
Troubleshooting table: symptom to cause
| Symptom | Most likely cause | Fix |
|---|---|---|
| Load fails with an unsupported architecture error | The llama.cpp build is too old | Upgrade and rebuild the engine |
| Loads fine but outputs garbage | Wrong chat template | Fix TEMPLATE or --jinja per the model card |
| Killed for running out of memory | You counted only the weights | Lower the context length or drop a quantization tier |
| Extremely slow per token | Part of the layers spilled to system memory | Use a smaller quant or a smaller model |
| Slow first token, then normal | Weights loading into memory on the first pass | Expected; warm it up |
| Tool calls do not format correctly | The template lacks a tool section | Add the tool section per the model card or change template |
| Repetitive output that never stops | Extreme sampling parameters | Adjust temperature and top_p, confirm an EOS token exists |
FAQ
Is there an official GGUF for hy4?
Which quantization should I use for a first hy4 deployment?
How do I work out how much memory I need?
Should I choose llama.cpp or Ollama?
The output quality is bad. Is the model weak?
Which route should a Mac user take for hy4?
Can I call Hunyuan through an API?
Related reading
- hy4 model page — specification, quantization matrix and memory math
- hy4-preview channel notes — what the 1M context channel is for
- Tencent Hunyuan family overview — every Hunyuan product line in one place
- hy4 model selection guide — sizes and where A13B fits
- Running hy4 with Ollama — pointing developer tools at a local endpoint
- Open WebUI with a local Hunyuan — a self-hosted chat interface
- VRAM calculation guide — the general memory math
- Local deployment playbook — the full path from hardware check to tooling
Sources
Quantization tiers and repository lists move as the community updates; this article reflects the state at sampling time. Memory figures are estimates, and your own measurements win.