Choosing an Embedding Model for Chinese RAG

When building a RAG system, developers typically spend significant time choosing an LLM but rarely evaluate the embedding model carefully. Yet 70% of RAG quality depends on retrieval, and the ceiling of retrieval quality is set by the embedding model — if it cannot map semantically similar texts to nearby positions in vector space, no LLM can rescue the results. This guide explains the role of embeddings in RAG, how to select open-source models for Chinese scenarios, and how to benchmark with your own data.

What embeddings do in RAG

The RAG pipeline: split documents into chunks → encode each chunk with an embedding model into vectors → store in a vector database. When the user asks a question, encode the query into a vector, find the "nearest" chunks in the vector store, and feed them into the prompt for the LLM to answer.

The embedding model's quality determines whether "nearest chunks" are truly semantically relevant. If two texts with the same meaning but different wording are mapped to far-apart vector positions, retrieval fails — the LLM receives irrelevant chunks and the answer is wrong.

Requirements for Chinese scenarios

Chinese embeddings face a unique challenge: semantic understanding. Chinese has no space-based tokenization, extensive polysemy, and high semantic density in idioms and abbreviations. English embedding models applied directly to Chinese usually underperform.

Three hard requirements for selection:

1. Chinese training data. The model's training corpus must include Chinese text. Check the model card's training data description to confirm.

2. Symmetric or asymmetric matching. Some embedding models optimize for asymmetric retrieval (query versus document, processed differently), and some for symmetric text-to-text matching. RAG usually needs asymmetric matching — user questions are short, document chunks are long.

3. Maximum sequence length. Embedding models have a maximum input limit (such as 512 tokens); text beyond it is truncated. If document chunks are longer, information in the excess is lost.

Mainstream open-source embedding models

Popular open-source families for Chinese scenarios (based on model card publications; no specific leaderboard scores cited in this article):

Local deployment routes

Route one: Ollama Embedding

The simplest route if Ollama is already running your LLM:

# Pull an embedding model (check Ollama for actual availability)
ollama pull nomic-embed-text

from langchain_community.embeddings import OllamaEmbeddings

embeddings = OllamaEmbeddings(
    base_url="http://127.0.0.1:11434",
    model="nomic-embed-text",
)

The advantage is zero additional deployment — if Ollama is already serving your LLM, embedding is just another API call. The limitation is that the model selection is constrained by the Ollama ecosystem; specialized Chinese embedding models may not be available there.

Route two: Sentence Transformers

More flexible: load any embedding model from HuggingFace directly:

# pip install sentence-transformers

from sentence_transformers import SentenceTransformer

# Load a Chinese-optimized embedding model (model name per HF releases)
model = SentenceTransformer("BAAI/bge-large-zh-v1.5")

# Encode texts
texts = ["What is a context window", "How to choose an embedding model"]
embeddings = model.encode(texts)
print(embeddings.shape)  # (2, 1024)

The advantage is unrestricted model choice — every embedding model on HuggingFace loads here, including the latest Chinese-optimized releases.

Route three: cloud API

If you do not want to manage local embedding models, cloud APIs are the simplest option, billed per call:

# Refer to the actual platform's embedding API
# EXAIV embedding availability follows the live /models catalog

Benchmarking with your own data

This is the most important step. Method:

1. Prepare a test set. Pick 20-50 chunks from your real documents, and handwrite 2-3 user questions for each.

2. Run retrieval tests. Encode all chunks with each candidate embedding model into the vector store, then search with the test questions and check whether the correct chunks are retrieved.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-zh-v1.5")

# Your document chunks
chunks = ["chunk 1 content...", "chunk 2 content...", "..."]
chunk_embeddings = model.encode(chunks)

# Test question
query = "A question a user might ask"
query_embedding = model.encode([query])

# Find most similar chunks
import numpy as np
similarities = np.dot(query_embedding, chunk_embeddings.T)[0]
top_3 = np.argsort(similarities)[-3:][::-1]
print("Top 3 most relevant chunk indices:", top_3)

3. Compare multiple models. Run the same test set through different embedding models and compare retrieval hit rates. The model with consistent high hit rates is your choice.

Quick decision table

Pulling the selection criteria together into a single reference:

Start with the simplest route that meets your language and quality requirements, benchmark with your own data, and upgrade only when a specific gap appears.

Dimension and speed tradeoff

Embedding vector dimensions affect storage and retrieval speed:

Most scenarios work well with 768 or 1024 dimensions. If document volume is below a million chunks, the speed impact of dimension is negligible — prioritize higher-dimensional models for better accuracy.

Common misconceptions

Misconception one: bigger embedding models are always better. Not necessarily. Larger embedding models have higher accuracy but slower speed, and encoding millions of documents becomes time-consuming. The criterion is "retrieval hit rate is high enough on your data" and "encoding speed is acceptable."

Misconception two: upgrading the LLM does not require upgrading the embedding. If you switch LLMs but retrieval stays the same, answer quality's ceiling is still set by retrieval. LLM and embedding are independent components — upgrading one does not upgrade the other.

Misconception three: English embeddings are sufficient for Chinese scenarios. English embedding models' semantic understanding of Chinese is typically worse than Chinese-optimized models. This is not "can it work" but "how much worse" — the gap in semantic retrieval tasks can be significant.

Misconception four: selection is one-time. Embedding models iterate continuously. If your RAG system's retrieval quality degrades after months of operation, it may not be the LLM's problem but new document types exceeding the current embedding model's capability range.

FAQ

Related reading

• RAG Knowledge Base Guide — the application context for embeddings
• Qwen GGUF Local Deployment Primer — local LLM deployment
• Buying Local Compute: A GPU Purchase Guide — hardware planning
• LangChain with EXAIV — full LangChain integration
• Pricing — live cloud pricing