Prompt Caching on EXAIV: Cutting the Input Cost of Repeated Context

In a chat application, the same block of text is sent over and over. A long system prompt, a policy document, a product catalogue excerpt, a retrieved knowledge base — it is identical on every single call, and every one of those calls pays full input price for it. In a long conversation the effect compounds, because the entire history is re-sent each turn.

Prompt caching addresses exactly this: the repeated prefix is stored and served from cache, at a lower price than fresh input tokens. The mechanism is simple, but getting the benefit requires the prompt to be structured in a particular way, and the benefit is measurable — which is what makes it worth verifying rather than assuming.

Why the input side dominates cost

Look at a typical multi-turn assistant request. Turn one sends the system prompt, some retrieved context, and the user's question. Turn two sends the system prompt, the retrieved context, the question, and the assistant's previous answer. Turn three sends all of that plus one more exchange.

The fixed part — system prompt plus retrieved context — is re-sent every time and is the same bytes every time. The growing part is the conversation history, which is genuinely new content on each turn.

That asymmetry is the whole opportunity. Fresh input tokens are billed at the full input rate. Cached input tokens are billed at a lower rate. If the fixed part is large and the conversation is long, the difference between serving it fresh and serving it from cache is substantial.

Structure the prompt for caching

The cache key is the request prefix. The rule follows directly:

[ system instructions ]  ← stable
[ tool definitions ]      ← stable
[ retrieved documents ]   ← stable within a session
[ conversation history ]  ← stable until the next turn
[ current user message ]  ← varies every request

Everything before the variable part is what can be cached. Put a timestamp, a request ID, or the current date in the system prompt and the prefix changes on every request, which means nothing is ever reused.

A concrete failure:

messages = [
    {"role": "system", "content": f"You are a support agent. Current time: {now()}"},
    {"role": "user", "content": question},
]

That first line changes the prefix on every call. Move the volatile part to the end:

messages = [
    {"role": "system", "content": "You are a support agent for Acme."},
    {"role": "user", "content": f"[request time: {now()}]\n{question}"},
]

Now the system prefix is stable and reusable.

Reading the cache counters

The usage object reports cached input separately:

usage = response.usage
input_tokens = usage.prompt_tokens
cached = getattr(
    getattr(usage, "prompt_tokens_details", None),
    "cached_tokens",
    0,
)

A useful derived metric:

hit_ratio = cached / input_tokens if input_tokens else 0

What to expect:

The interesting case is the multi-turn conversation: as the history grows, the cached fraction should grow too. If it stays flat, the prefix is changing.

What invalidates a cache

The first three are structural and worth designing around. The fourth is a business reality: if your retrieved context genuinely differs every call, the document block is not cacheable, and moving it after the user message does not help because the user message must come last.

What to measure

Three numbers answer whether caching is doing anything:

1. Cache hit ratio — cached input tokens over total input tokens.
2. Cached tokens per request — the absolute volume served from cache.
3. Cost per successful request — the number that shows whether the change mattered.

Track them per feature. A high hit ratio on one endpoint and zero on another usually means the prompts are structured differently, not that the platform is behaving inconsistently.

Caching and long conversations

A long conversation is the case where caching matters most, and it is also the case where the cache key behaves in a way that is easy to misjudge. As the conversation grows, the history prefix grows with it — and that larger prefix is exactly what can be reused on the next turn.

Two properties make this work:

• The history is append-only. Each turn adds to the end, so the prefix from the previous turn is unchanged. A cache entry from turn ten covers turn eleven.
• The volatile part is at the very end. The newest user message changes, and everything before it does not.

This means the cached share of a long conversation grows turn by turn, which is visible in the usage counters: an early turn has a small cache hit, and a late turn has a large one. If the hit ratio stays flat across a long conversation, the prefix is changing for some other reason — usually a timestamp, an identifier, or a re-serialized tool definition earlier in the prompt.

The practical consequence is that the most valuable place to look for caching wins is the system prompt and the retrieved context, not the conversation itself. The history is already cacheable by construction; the fixed blocks above it are what decide whether the cache is worth having at all.

Where caching pays and where it does not

Pays:

• long system prompts re-sent on every call;
• multi-turn conversations, where the history is stable across turns;
• repeated identical requests from a batch job;
• large tool definitions that never change.

Does not pay:

• one-off requests with a short prompt;
• workflows where retrieved context is regenerated per call;
• prompts with a volatile field near the front;
• workloads small enough that the cost was never significant.

For a workload in the second group, the fix is not caching but reducing the prompt: a 3,000-token system prompt that could be 300 tokens is cheaper at any cache hit rate.

Practical prompt layout

A layout that tends to cache well:

STATIC_SYSTEM = "You are a support agent for Acme. Follow the policy below.\n\n" + POLICY

def build_messages(question, history, now_iso):
    return [
        {"role": "system", "content": STATIC_SYSTEM},        # stable
        *history,                                            # stable within the turn
        {"role": "user", "content": f"[{now_iso}] {question}"},  # volatile, last
    ]

Rules that fall out of this:

• Keep static content in a module-level constant, not something rebuilt per request.
• Do not put request metadata, identifiers, or timestamps in the system prompt.
• Keep the conversation history in a stable order and do not re-serialize it differently between turns.
• Put the newest information at the end of the prompt, where it is both cache-friendly and closest to the generation point.

A caution about sensitive content

A cached prefix is a stored copy of that content on the platform side. If a prefix contains confidential material, treat it as retained data and confirm how it is handled before relying on caching for it. The practical guidance is the same as for any request payload: do not put secrets in prompts, and be deliberate about what is stable enough to persist.

Measuring the win

Caching is easy to implement and easy to believe in without measuring. Three numbers tell you whether it is working, and they should be tracked before and after the change, on the same traffic.

Hit ratio. Cached input tokens divided by total input tokens, per feature. This is the fastest signal: a hit ratio near zero means the prefix is not stable, no matter how the code looks.

Cached tokens per request. The absolute volume served from cache. A high ratio on small requests saves little in absolute terms, while a modest ratio on a large prompt can save a lot.

Credits per successful request. The number that decides whether the change was worth making. Compare it over the same period, on the same traffic, before and after.

If the third number did not move, caching is not helping — regardless of what the hit ratio says. That happens when the prompt is small, when the traffic is one-off, or when the cached rate is close enough to the fresh rate not to matter.

Caching checklist

• Stable content comes first, variable content last.
• No timestamp, request ID, or other volatile value sits in the system prompt.
• Tool definitions are stable across requests.
• Cached token counters are read from the usage object and recorded.
• Hit ratio is tracked per feature.
• Cost per successful request was compared before and after the change.
• The prompt was shortened as well as reordered, where length was the real problem.
• The content of any cached prefix was reviewed for sensitivity.

Further reading

• Token Usage and Cost Tracking — where the cache counters are recorded.
• Chat Completions Parameters Reference — the request fields that shape the prompt prefix.
• GPT Model Selection Guide — the tier decision that sets the price caching discounts against.

FAQ

Related reading

• Qwen Streaming and Tool Calling
• Qwen Cost Optimization
• Context Engineering Guide