hy4 Local RAG in Practice: Wiring a 4B Model into a Knowledge Base and Vector Search
hy4 Local RAG in Practice: Wiring a 4B Model into a Knowledge Base and Vector Search
Whether a 4B model can do RAG depends entirely on what you ask it to do. The answer is: for the lightweight stages of a retrieval pipeline it is plenty; for long-form synthesis it will fall over. Hunyuan hy4 fits the first role well — 4.22 billion dense parameters, a native 256K context, and a Q4_K_M build that fits in 2.6GB. This article gives you a complete local setup: a local vector store, bge-m3 embeddings, hy4 handling rewriting and generation, plus the five failure modes you will actually hit.
The 30-second version
hy4 has three high-value jobs in RAG: query rewriting (turning a colloquial question into retrieval terms), intent routing (deciding which knowledge base to hit), and short answer generation (a grounded answer under 300 words from retrieved snippets). Leave multi-document synthesis and multi-hop reasoning to a larger model. For embeddings use an open model such as bge-m3, since the EXAIV cloud catalog carries no embedding models today.
Define the split first: what a 4B should and should not do
A RAG pipeline has at least two roles: retrieval that finds material, and generation that organises it into an answer. These demand very different things from a model:
| Stage | What it needs from the model | Is hy4 suitable |
|---|---|---|
| Query rewriting / expansion | Understand the question, emit keywords | Very suitable |
| Intent classification / routing | Emit one label | Very suitable, 0.5B would even do |
| Summarising / extraction | Cover the key points of a passage | Suitable |
| Grounded short answers | Stay faithful, avoid invention | Suitable with strict prompting |
| Multi-document long synthesis | Long-range reasoning and consistency | Not suitable, use a larger model |
| Multi-hop reasoning | Intermediate reasoning steps | Not suitable |
Why small models suit the front of the pipeline
The front of the pipeline produces short text or labels, where errors are cheap and verifiable. The back produces long text, where errors are expensive and hard to trace back. Put the 4B model in front and a bad rewrite still has recall to fall back on. Put it at the end and every hallucinated sentence lands in front of the user.
A complete local setup
Step 1: start the local inference server
./llama-server -m ./hunyuan4b/Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 \
--ctx-size 32768 --jinja
The exposed endpoint is a standard OpenAI-compatible API, so downstream code points at http://127.0.0.1:8080/v1.
Step 2: chunk the documents
from pathlib import Path
import re
def load_docs(folder: str) -> list[str]:
chunks = []
for p in sorted(Path(folder).glob("**/*.md")):
text = p.read_text(encoding="utf-8")
# split by heading level first, then by length, keeping the heading as a prefix
sections = re.split(r"\n(?=#{1,3} )", text)
for sec in sections:
if len(sec) < 200:
continue
for i in range(0, len(sec), 1200):
piece = sec[i:i + 1200]
if len(piece.strip()) > 50:
chunks.append(sec.split("\n")[0] + "\n" + piece)
return chunks
chunks = load_docs("./docs")
print(len(chunks))
First rule of chunking: never cut a semantic unit in half
Fixed-size character splitting slices a table or an API description in two, and the retrieved snippet then reads as a fragment the model has to guess at. Split by heading hierarchy first, fall back to length, and prepend the heading — the heading is the best retrieval anchor you have.
Step 3: embed locally
from sentence_transformers import SentenceTransformer
embed_model = SentenceTransformer("BAAI/bge-m3")
vectors = embed_model.encode(chunks, normalize_embeddings=True, batch_size=32)
bge-m3 is one of the most downloaded open multilingual embedding models, with roughly 29.5 million downloads in the last 30 days, and it handles mixed Chinese and English plus long inputs.
Why not a hosted embedding service
First, the EXAIV cloud catalog has no embedding models today. Second, embedding traffic is high volume and low value per request, so hosting it costs more than it saves. Third, embedding models are usually a few hundred megabytes to around a gigabyte, so running one locally costs almost nothing operationally and is usually faster end to end.
Step 4: recall, then rerank
import numpy as np
query_vec = embed_model.encode([query], normalize_embeddings=True)[0]
scores = np.dot(vectors, query_vec)
top_idx = np.argsort(-scores)[:20]
# rerank the candidates with a cross-encoder
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
pairs = [[query, chunks[i]] for i in top_idx]
rerank_scores = reranker.predict(pairs)
final_idx = [top_idx[i] for i in np.argsort(-rerank_scores)[:4]]
Reranking is the highest-leverage step
Vector recall is a coarse filter: cheap, fast, wide. A cross-encoder reranker is a precise filter: slower, pricier, far better ordered. In a lot of RAG systems the answer quality bottleneck is retrieval ordering, not the generator. Spending effort turning top-20 into a clean top-4 usually beats swapping a 4B generator for a 7B one.
Step 5: rewrite and answer with hy4
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
def rewrite(query: str) -> str:
r = client.chat.completions.create(
model="local-model",
messages=[
{"role": "system", "content": "Rewrite the user question as 3 keyword phrases for vector search, one per line, no explanation."},
{"role": "user", "content": query},
],
temperature=0.1,
)
return r.choices[0].message.content
def answer(query: str, context_chunks: list[str]) -> str:
context = "\n\n---\n\n".join(context_chunks)
r = client.chat.completions.create(
model="local-model",
messages=[
{"role": "system", "content":
"You are a knowledge base assistant. Answer only from the supplied material. "
"If the material does not cover it, say so plainly. "
"Keep the answer under 300 words and mark each claim with [snippet N]."},
{"role": "user", "content": f"Material:\n{context}\n\nQuestion: {query}"},
],
temperature=0.2,
)
return r.choices[0].message.content
rewritten = rewrite(query)
print(answer(query, [chunks[i] for i in final_idx]))
Hard-code three constraints in the prompt
Answer only from the material, say so when the material does not cover it, and cite snippet numbers. Those three lines cut hallucination on a 4B model dramatically. Small models do not decide on their own whether to invent, but they can be constrained by instruction.
The five traps you will hit
Trap one: chunking breaks semantics
The symptom is a retrieved snippet that reads as a fragment and an answer containing half a sentence. Fix it by splitting on heading hierarchy, keeping the heading as a prefix, and re-splitting long paragraphs on sentence boundaries.
Trap two: recall without reranking
The symptom is that the correct document was retrieved but landed at position 12 and got cut by the top-4 filter. Fix it by raising the recall count, adding a reranker, or running several rewritten queries and merging the result lists.
Trap three: no constraint in the prompt
The symptom is a confident answer built from training knowledge even though the material does not contain it. Fix it with an explicit "not in the material" instruction, snippet citations, and a temperature below 0.2.
Trap four: treating retrieval as question answering
The symptom is a user asking how to use a feature and getting a product blurb describing what it is. Fix it by classifying the question first (fact lookup, how-to, troubleshooting) and using a different prompt template per class.
Trap five: overloading the context
The symptom is hy4 failing to find the key sentence in a wall of material and starting to repeat itself. hy4 natively supports a 256K context, but fitting a window is not the same as using it well. Four to eight snippets per call is the stable range; compress beyond that.
When a 4B generator is the wrong tool
| Situation | Better choice |
|---|---|
| Synthesising across more than five sources | A larger cloud model |
| Producing a long report or proposal | A larger cloud model |
| Retrieval quality is poor and material never surfaces | Fix retrieval, do not swap generators |
| High-volume Q&A where latency matters | A lightweight model such as hy4 |
| Long documents that need long summaries | A stronger long-context model, or summarise in layers first |
FAQ
Is a 4B model good enough for RAG?
How much context does hy4 support?
Which embedding model should I use?
Do I need a reranker?
How do I reduce made-up answers?
How do I connect a local hy4 endpoint to an existing app?
Related reading
- hy4 model page — specification, quantization matrix and memory math
- hy4-preview channel notes — what the 1M context channel is for
- hy4 GGUF local deployment — the full steps to get the server running
- Running hy4 with Ollama — pointing developer tools at a local endpoint
- Open WebUI with a local Hunyuan — a self-hosted chat interface
- Context engineering guide — treating the window as a budget
- Embedding model selection guide — choosing between bge, gte, jina, and e5
- hy4 model selection guide — sizes and where A13B fits
Sources
Download counts and parameter specifications follow the live official repositories, sampled in November 2026.