Context Engineering in Practice: Putting the Right Information in the Window at the Right Time

Context windows keep growing, but "it fits" does not mean "it works." Dump 100K of material into a request and the model tends to lose information in the middle and have its attention diluted by irrelevant content. Context engineering answers: what to put in a finite window, in what order, and when to refresh it. It is the design layer above RAG, prompting, and memory management — and the deciding factor in whether agent systems run reliably.

What context is made of

A single request context usually contains five blocks, all competing for the window budget:

The key insight: the window is a budget, not a container. Every block must answer "is this necessary for the current step."

Three typical failure modes

1. Information dilution. Retrieval returns ten chunks, only two are truly relevant, and the other eight thin out attention. Fix: tighten top-k, add a similarity threshold, re-rank then truncate.

2. Lost in the middle. In long contexts, information placed in the middle is easier to overlook than information at the beginning and end. Fix: put critical constraints in the system prompt (start) and adjacent to the question (end); summarize long material structurally before placing it in the middle.

3. History bloat. Every turn carries the full history, the window fills with old content, and new information cannot get in. Fix: keep the recent N turns verbatim, summarize older ones; or retrieve history fragments by topic instead of carrying everything.

Compression and layering strategies

Summarize: compress long material and old dialogue into key points, keeping conclusions and numbers while discarding process text.

Layer: stable rules go in the system prompt (present every call, cacheable); dynamic material goes in retrieved chunks (fetched as needed); temporary intermediate results go in tool returns (discarded after use).

Retrieve on demand: do not fetch all candidates upfront — let the model decide which kind of information it needs, then fetch the matching chunks (agent-style retrieval).

Structure: organize context with headings, lists, and tables; models locate structured content more reliably than one free-form block.

# Minimal history compression: keep recent 3 turns, summarize older
def build_context(history, retrieved, system_prompt):
    recent = history[-3:]
    older_summary = summarize(history[:-3]) if len(history) > 3 else ""
    parts = [system_prompt, older_summary, *recent, "Relevant material:\n" + retrieved]
    return "\n\n".join(p for p in parts if p)

Relation to RAG and prompt engineering

They are not alternatives: prompt engineering writes the system prompt, RAG produces retrieved chunks, and context engineering decides their order, quota, and refresh timing. When a RAG system underperforms, do not rush to swap the embedding model — the problem is often contextual: too many chunks, uncompressed history, poor ordering.

One complete example: context design for support Q&A

Applying the principles above to a concrete case — a support Q&A system over an internal knowledge base.

System prompt (stable prefix): role, tone, refusal rules, output format — roughly 200 tokens, unchanged every call, cache-friendly.

Retrieved chunks (dynamic): after the user asks, fetch the 3 most relevant chunks, order by relevance, and separate them with headings. Do not fetch ten low-relevance chunks and dilute attention.

History (compressed): recent 3 turns verbatim, older ones compressed into one sentence. On follow-up questions, rewrite "what the user is actually asking" into a standalone query and retrieve on that, instead of matching the entire history.

Output constraints (at the end): place "answer only from the material; say so if insufficient" adjacent to the question, not buried mid-context.

A request context becomes "stable prefix + 3 chunks + 3 turns + ending constraints" rather than "system prompt + all material + all history". The difference in answer quality is usually striking — and it requires no model swap. This is the practical payoff of treating the window as a budget: you spend tokens where they change the answer, and stop spending them where they do not. Teams that adopt this discipline often find they can also lower cost and latency at the same time, because a shorter context is both cheaper and faster to process. Combine it with a fixed eval set to confirm each structural change actually helps rather than merely looking tidier. Start by measuring the current average context length per request, then trim one component at a time and re-run the eval. Keep the numbers: they are the evidence that the redesign worked.

How context engineering affects cost and latency

Every token in the window is processed on every call, so context size drives both cost and latency directly. A request with 20K tokens of mostly irrelevant material costs far more and responds more slowly than one with 4K tokens of well-organized material — often with worse answers. Trimming the window is one of the few optimizations that improves quality, cost, and speed at the same time.

Practical steps: measure the average context length of your real traffic, then set a budget per component (for example system prompt 300 tokens, retrieved chunks 1500 tokens, history 800 tokens). Enforce the budget in code rather than hoping prompts stay short. Put stable content in a cacheable prefix so repeated calls do not reprocess it. Re-measure after each change: use a fixed eval set to confirm quality did not regress while you cut tokens. Combine with batch inference for offline workloads, and with a fixed eval set for regression, and the whole pipeline becomes measurable end to end — which is the point of treating context as an engineering surface rather than an art.

A short checklist before shipping

1. Measure your current average context length per request.
2. Assign each component a token budget and enforce it in code.
3. Move stable content into a cacheable prefix.
4. Trim retrieval to 3-5 re-ranked chunks with a similarity threshold.
5. Compress history: recent turns verbatim, older ones summarized.
6. Place critical constraints at the start and immediately before the question.
7. Run a fixed eval set before and after each change; keep the numbers.
8. Watch for silent degradation after upstream model updates — monthly regression on cloud APIs.

Context engineering is not a one-time redesign. As traffic patterns, document collections, and model versions change, the right budget changes too. Treat the eval set and the per-component measurements as the feedback loop: they tell you when a previously sensible layout has started to waste tokens or lose information. This is the same evidence-first discipline described in the evaluation guide, applied to the structure of every request rather than to model selection. Done consistently, it compounds: shorter contexts reduce cost and latency on every call, while the eval set protects the quality gains.

FAQ

Related reading

• RAG Knowledge Base Guide — retrieval and context together
• LLM Evaluation Methodology — regression with a fixed eval set
• Qwen Cost Optimization — controlling token consumption
• VRAM Calculation in Practice — concurrency VRAM ceiling
• Pricing — live cloud pricing