Building a RAG Knowledge Base with Open Chinese Models

RAG (Retrieval-Augmented Generation) is one of the most valuable applications for open Chinese models: feed your documents, codebase, or internal knowledge into the model so it answers based on your data — rather than guessing from generic training data. This guide gives a complete RAG solution using open Chinese models: embedding selection, vector store choices, document chunking strategy, and both a local route (Ollama Embedding + llama.cpp) and a cloud API route.

The four stages of RAG

RAG is not "stuff documents into the model." It is a pipeline:

Embedding selection

Selection criteria: multilingual support (essential for Chinese scenarios), vector dimension (affects storage and retrieval speed), and model size (affects embedding speed).

The local route uses Ollama's Embedding most simply:

# Pull an embedding model (community-maintained; check Ollama for availability)
ollama pull nomic-embed-text

Vector store selection

Start with Chroma for personal and small teams — pure Python, zero external dependencies, one line to store vectors.

Full implementation: 30 lines of LangChain

A minimal runnable RAG pipeline using local models:

# pip install langchain langchain-community chromadb

from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain.text_splitter import RecursiveCharacterTextSplitter

# 1. Document chunking
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,       # each chunk roughly 500 characters
    chunk_overlap=50,     # 50 characters overlap between chunks
)
docs = splitter.split_documents(your_documents)

# 2. Embedding + store in vector DB
embeddings = OllamaEmbeddings(base_url="http://127.0.0.1:11434", model="nomic-embed-text")
vectorstore = Chroma.from_documents(docs, embeddings)

# 3. Retrieval + generation
llm = Ollama(base_url="http://127.0.0.1:11434", model="your-model")
query = "Your question here"
results = vectorstore.similarity_search(query, k=3)  # top 3 most relevant chunks
context = "\n\n".join([r.page_content for r in results])
prompt = f"Answer the question based on the following content.\n\n{context}\n\nQuestion: {query}"
answer = llm.invoke(prompt)
print(answer)

Cloud route: the EXAIV endpoint

If you do not want to manage local models, use EXAIV's OpenAI-compatible endpoint for the generation stage:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://exaiv.com/api/ai/v1",
)

# Retrieved chunks fed into the system prompt
resp = client.chat.completions.create(
    model="kimi-k3",  # per the live /models catalog
    messages=[
        {"role": "system", "content": f"Answer the question based on the following content. If the content is insufficient, say so explicitly.\n\n{context}"},
        {"role": "user", "content": query},
    ],
)
print(resp.choices[0].message.content)

Advanced RAG techniques

Once the basic pipeline works, several techniques improve retrieval quality:

1. Re-ranking. After the vector store returns candidate chunks, a re-ranking model (like a cross-encoder) re-scores and re-orders them by relevance to the query. This catches cases where vector similarity is high but actual semantic relevance is low.

2. Query expansion. Before retrieval, an LLM rewrites the user's query into multiple variations, then searches for all of them. This improves recall when the user's phrasing does not match the document's vocabulary.

3. Metadata filtering. Store metadata (source, date, document type) alongside each chunk, then filter by metadata before vector search. This narrows the search space and improves precision for structured document collections.

4. Hybrid search. Combine vector similarity search with keyword (BM25) search, merging results by weighted rank. This catches both semantic matches and exact keyword matches that vector-only search might miss.

None of these require changing your LLM — they improve the retrieval layer that feeds the prompt.

Production deployment considerations

Moving from prototype to production introduces several considerations:

Persistence. Chroma's default in-memory mode loses vectors on restart. Switch to a persistent backend (Chroma with SQLite, or Qdrant) once you move past prototyping.

Incremental updates. When documents change, you need to re-embed only the changed chunks, not the entire corpus. Design your pipeline to track which chunks came from which source documents.

Evaluation. Build a small evaluation set (question + expected source chunk + expected answer) and run it after every pipeline change. Without evaluation, retrieval quality drifts silently.

Monitoring. Track retrieval precision (did the right chunks get retrieved?) separately from generation quality (was the answer correct?). They fail for different reasons.

Cost management. In cloud mode, RAG calls consume tokens for both retrieved context and generation. Monitor context size and trim retrieved chunks to control cost.

Document chunking strategy

Chunking granularity directly affects retrieval quality:

Common RAG pitfalls

These are the mistakes that most often cause RAG projects to underperform. Each one is easy to avoid with a small amount of upfront planning, but surprisingly common in real deployments:

1. Skipping evaluation. Without a fixed question set scored against expected sources, you cannot tell whether a change improved or degraded retrieval. Build the evaluation set first, even if it is just ten questions.

2. Using the same model for embedding and generation. They serve different purposes. A lightweight embedding model with a strong generation model usually outperforms using one model for both.

3. Ignoring document quality. RAG cannot fix poorly structured source documents. Clean your documents (remove boilerplate, normalize formatting) before chunking — the quality of what goes in determines the quality of what comes out.

4. Over-complicating the prompt. The retrieved context already provides the information; the prompt should instruct the model to use it and acknowledge gaps, not add layers of instruction that confuse the model. Simple, direct instructions work better than complex multi-paragraph system prompts. A good baseline: tell the model to answer based only on the content below, and to say it cannot answer if the content does not contain the answer.

5. Not handling "I don't know." When retrieval returns no relevant chunks, the model should say so rather than hallucinate. Add an explicit instruction: if the context does not contain the answer, say you cannot answer based on the provided content. This single instruction prevents the most damaging class of RAG failures — confidently wrong answers — and builds user trust in the system.

Local versus cloud tradeoff

Sensitive data → local route (data never leaves the machine); flagship quality needed → cloud route (kimi-k3 and other flagship models); mixed → local Embedding + cloud LLM (Embedding carries no sensitive info, LLM called from EXAIV).

FAQ

Related reading

• LangChain with EXAIV — the full LangChain integration tutorial
• The Complete Local Deployment Playbook — the full local deployment flow
• Which Qwen Model to Use — model selection
• Chinese Open Weights vs US Flagships — local versus cloud
• Pricing — live cloud pricing