LLM Evaluation Methodology: Building a Trustworthy Eval for Your Own Use Case
LLM Evaluation Methodology: Building a Trustworthy Eval for Your Own Use Case
You change a model, revise a prompt, upgrade a version — how do you know whether it got better or worse? Public benchmark scores are mostly useless here, because their test data is usually completely different from your real scenario. This guide explains how to build an evaluation system for your own business: the eval set, scoring methods, A/B comparison, and regression testing. With it, every model change becomes evidence-based instead of a gut feeling.
The short version
The core of evaluation is not picking a score but building a fixed eval set: sample 30-100 representative inputs from real workloads → define what a good answer looks like → run the same set before and after every change → score by humans or LLM-as-judge → compare results before deciding. The eval set itself needs versioning, just like code.
Why public scores are not enough
Public benchmarks like MMLU and MTEB are valuable, but they have three inherent limits:
- Different data distribution: they test general knowledge or academic problems, while your workload may be internal documents, specific formats, and domain jargon
- None of your constraints: output format, tone, compliance requirements, and refusal boundaries are not measured
- Optimizable targets: models may have seen similar items in training, so high scores do not prove generalization
The correct use of public scores is narrowing the candidate list; final selection must use your own eval set.
An eval set is an asset, not a one-off script
A good eval set is reused repeatedly: model swaps, prompt edits, engine upgrades, temperature changes — all run the same items. It grows with your business and needs versioning like code. Half a day of setup saves time on every later decision.
Step 1: build the eval set
1. Sample from real workloads. Pick 30-100 representative inputs from production logs or stakeholder requirements, covering:
| Category | Suggested share | Example |
|---|---|---|
| High-frequency typical tasks | 50-60% | The most common everyday questions |
| Edge / hard cases | 20-30% | Long inputs, ambiguity, multiple constraints |
| Must-refuse / boundary | 10-20% | Should-not-answer, insufficient information |
2. Define what a good answer means for each. Not one frozen gold answer (most tasks have none), but checkpoints:
- Are facts correct
- Are the required points covered
- Does the format comply (JSON fields, length, tone)
- Does it hallucinate when it should not
3. Store as structured data. One sample contains id, input, checkpoints, and a category label. JSONL or CSV both work; the important part is stable script readability.
{
"id": "case-001",
"input": "Rewrite this product description as an image generation prompt",
"checkpoints": ["Includes subject and style descriptors", "Single-paragraph output", "Adds no numbers absent from the source"],
"category": "high_frequency"
}
Step 2: choose scoring methods
| Method | How | Fits | Limit |
|---|---|---|---|
| Human scoring | People score against checkpoints | Quality-sensitive, small sets | Slow, costly, subjective |
| LLM-as-judge | A strong model scores against a rubric | Medium scale, fast iteration | Judge has its own bias |
| Automatic metrics | Exact match, JSON validation, regex | Structured output | Cannot judge open-ended quality |
In practice, combine them: automatic metrics as a floor for structured tasks, LLM-as-judge for first-pass screening on open-ended tasks, and human spot-checks on samples where the judge disagrees.
Two disciplines for LLM-as-judge
(1) Use a strong model from a different family than the model under test to avoid self-preference bias. (2) Write the scoring rubric into a fixed file and do not reword it each run. Judge scores are for relative comparison, not absolute quality conclusions — a judge can also be swayed by surface fluency. Human spot-checks are a necessary safety net.
Step 3: A/B comparison
When comparing two models or two prompts, discipline matters more than tooling:
- Fix the same items and parameters (temperature, max_tokens, system prompt — everything except the variable under test)
- Change exactly one variable — swapping the model and editing the prompt at once makes results impossible to attribute
- Blind scoring: shuffle A/B output order before giving it to scorers to avoid position bias
- Record per-item results, not just averages — A may be clearly better on long inputs while B is slightly better on short ones
- Look at the distribution, not only the mean: when averages are close, compare the severity of failures — one severe hallucination matters more than ten minor style differences.
# Minimal A/B skeleton: run two configurations on the same items and save results
import json
cases = [json.loads(l) for l in open("eval_set.jsonl")]
results = []
for case in cases:
for variant in ["model_a", "model_b"]:
out = call_model(variant, case["input"]) # your call wrapper
results.append({"id": case["id"], "variant": variant, "output": out})
json.dump(results, open("run_001.json", "w"), ensure_ascii=False, indent=2)
Step 4: regression testing workflow
Wire evaluation into the change process:
- Model swap / version upgrade: run the full eval set, compare with the previous baseline
- Prompt edit: run the eval set plus the subset relevant to that prompt
- Engine / quantization tier change: run the eval set and record latency and VRAM too
- Decision rule: key checkpoints must not regress; small average fluctuations are judged by distribution
The value of regression is catching silent degradation: an upstream model update produces no errors but changes output style — only a fixed eval set detects it. Cloud API models change as the platform updates them, so run regression monthly; for local models, run it before every upgrade.
Evaluation is not one-time
The eval set grows with the business: add new cases, keep old ones. Archive each run result with date, model/version, parameters, and scores. Three months later, when you want to reconstruct "why we picked this model," those records are the answer.
A worked example: choosing between two models
Suppose you are comparing two candidate models for an internal document-summary feature. Your eval set has 40 items: 24 typical, 10 long or ambiguous, 6 must-refuse boundaries.
- Run both models on all 40 items with temperature 0 and identical system prompts.
- Score checkpoints per item: fact coverage, format compliance, no invented numbers.
- Compute per-category pass rates, not just an overall average. Model A may win on typical items while Model B handles long inputs substantially better.
- Inspect the severe failures. If Model A produces three fabricated figures while Model B produces none, that dominates a small average-score difference — fabricated numbers are unacceptable in an internal summary tool even if the average looks close.
- Decide with a documented rule: key checkpoints must not regress, and the category that dominates real traffic gets the higher weight.
- Archive the run: date, model IDs, parameters, per-item scores, and the decision. Three months later, this record is the answer to "why did we choose this model".
Notice that no public benchmark appears anywhere in this process. The eval set, the checkpoints, and the decision rule are all specific to your product — which is exactly why the conclusion is trustworthy.
Common misconceptions
Myth one: bigger eval sets are better. 30 high-quality cases covering edge scenarios carry more information than 500 duplicate high-frequency ones. Secure coverage first, then scale.
Myth two: only the average matters. Averages hide distributions: one severe error matters more than ten small fluctuations — grade failures by severity.
Myth three: edit the eval set along with the prompt. The eval set is the ruler; do not change items to make scores look good. If you must change it, record the version change explicitly and rebuild the baseline.
Myth four: treat LLM-as-judge conclusions as truth. It is a relative comparison tool; calibrate with human spot-checks, especially in high-risk scenarios.
Operating the eval over time
An eval set is not a one-off artifact — it decays as your product changes. Three habits keep it trustworthy across quarters:
- Append, do not overwrite. When a new failure mode appears in production, add a case that reproduces it and keep the old cases intact. A shrinking or silently rewritten eval set hides regressions instead of catching them.
- Version the eval alongside the prompt. Record the eval-set hash, the prompt template, and the model ID in the same result row. Without that triple, a score change is uninterpretable: you cannot tell whether the model, the prompt, or the questions moved.
- Re-review the rubric. Human and LLM-as-judge scoring drift as reviewers change. Re-anchor on a small fixed subset of gold cases every few weeks so the scale keeps the same meaning.
Cost matters too. Judging every candidate on the full set is rarely necessary: a cheap smoke subset on every change plus a full run before release catches most regressions at a fraction of the spend. Log tokens and latency per eval run the same way you log them per production request — an eval harness that is too expensive to run is a harness nobody runs.
FAQ
How large should the eval set be?
Is LLM-as-judge trustworthy?
Cloud models update — how do I prevent degradation?
What matters in A/B comparison?
Should eval data be separate from fine-tuning data?
Where is pricing?
Related reading
- Embedding Model Selection Guide — empirical testing for retrieval
- RAG Knowledge Base Guide — a typical scenario that needs evaluation
- Model Version Management — upgrade and rollback flow
- Qwen Cost Optimization — evaluation and cost
- Production Readiness Checklist — pre-launch checks