Token Usage and Cost Tracking on EXAIV: Reading Usage Data and Turning It Into a Budget

The response body of every request contains the numbers you need to attribute spend: how many tokens went in, how many came out, and how many were served from cache. Once those numbers reach your own logs and dashboards, cost stops being a monthly surprise and becomes a per-feature line item you can act on.

The gap between the two states is usually small work — reading a usage object, normalizing it, and storing it in a place you already query. This guide covers what the usage data contains, how to convert it into credits, how to attribute it to features, and how to notice a spike while it is still small.

Reading the usage object

A non-streaming response includes usage at the top level:

{
  "id": "chatcmpl-...",
  "model": "gpt-6-sol",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "..."}, "finish_reason": "stop"}
  ],
  "usage": {
    "prompt_tokens": 1840,
    "completion_tokens": 226,
    "total_tokens": 2066
  }
}

In Python and Node.js the same object is available directly:

usage = response.usage
print(usage.prompt_tokens, usage.completion_tokens, usage.total_tokens)

const { prompt_tokens, completion_tokens, total_tokens } = response.usage;

When streaming, usage may not be present in the chunks. Send the request with stream_options requesting usage, or issue one non-streaming call when you need an exact record — do not assume the streaming path gives you the same accounting.

From tokens to credits

Credits are computed from token counts against the rates published for each model in the model directory. The structure is simple:

Two consequences follow, and both are worth checking early:

• Prompt length dominates cost in chat workloads. A system prompt, a long history, and a retrieved document all land in the input side of every single turn. Trimming a thousand tokens from a system prompt saves a thousand tokens on every request, forever.
• Output is where the runaways happen. A request that was supposed to return one paragraph and returns ten pages is a cost incident, and only the output counter reveals it.

Attributing usage to a feature

Total spend is not actionable. "You spent X this month" does not tell you what to fix; "the summarization feature spends 60% of your credits and returns one paragraph per call" does.

The reliable way to attribute is to decide at call time and carry the label with the request:

import time, uuid

def call_model(feature, messages, **kwargs):
    request_id = str(uuid.uuid4())
    started = time.time()

    response = client.chat.completions.create(
        model="gpt-6-sol",
        messages=messages,
        **kwargs,
    )

    usage = response.usage
    record_usage(
        request_id=request_id,
        feature=feature,             # "summarize" | "classify" | "chat" | ...
        model=kwargs.get("model", "gpt-6-sol"),
        input_tokens=usage.prompt_tokens,
        output_tokens=usage.completion_tokens,
        cached_tokens=getattr(usage, "prompt_tokens_details", None)
                       and getattr(usage.prompt_tokens_details, "cached_tokens", 0) or 0,
        latency_ms=int((time.time() - started) * 1000),
        status="ok",
    )
    return response

Two properties make this table useful later. It is keyed by a request ID you generated, so you can join it to your own application logs. And it carries the feature label, so spend can be sliced by feature without any natural-language guessing.

The metrics worth watching

Raw token totals are a lagging indicator. These ratios tell you what to fix:

Two of these deserve special attention. Cache hit ratio is a direct measure of avoidable spend: if the same large prefix is sent uncached on every call, you are paying full input price for identical content. Retries per request multiplies everything — a request that succeeds on the third attempt costs three times what a clean first attempt costs.

Spotting anomalies early

Anomaly detection does not need machine learning. Most real cost incidents have a simple shape: a step change, a new feature that nobody announced, or a specific input that produces a much larger output than expected.

Useful, low-effort alerts:

• Daily credits over budget. A hard threshold catches runaway loops immediately.
• Change versus trailing baseline. Alerting on "today is 40% above the same weekday last week" catches a regression that a fixed threshold misses.
• Tokens per request above the p95. A single pathological prompt is often the cause, and this points straight at it.
• Error rate spike with retries. Failures and retries correlate directly with spend.
• A new model ID appearing. Traffic drifting to a stronger tier is usually a configuration change, and it is worth knowing about.

What to store

A minimal, useful schema:

Keep the raw counts rather than only the derived credit amount. The rate table changes, and a table of token counts can be re-costed against any future rate; a table of credits cannot.

Verifying the numbers

Once the pipeline exists, check it against reality:

1. Compare a single day of your records against the platform console. A mismatch means a field is being dropped or misattributed.
2. Recompute one request by hand from its token counts and the published rate. This validates the rate source and the rounding.
3. Check that retries are counted once per attempt, not once per logical request. Undercounting here is the most common reporting bug.
4. Verify the cache counter is being read from the right field for the models you use.

A cost dashboard that is off by a constant factor is still useful for trends. A dashboard that is off by an unpredictable factor is worse than none, because it destroys trust in the numbers.

Reporting spend by tenant or customer

Once the usage table exists, the question that follows is how to slice it. A single total tells you that the bill is what it is. A breakdown by customer, tenant, or product area tells you what to change.

Two things have to be in place:

A stable identifier. Every request needs a label that maps to a customer, a team, or a feature. It has to come from your own request context — the authenticated caller, the workspace, the route — and not from a value the caller can freely set in the payload.

A dimension you can defend. If several end users share one account, decide whether spend is attributed to the account or to the individual, and be consistent. Two dashboards that attribute the same spend differently will eventually be used in an argument.

Once the slices exist, the useful question changes from "how much did we spend" to "which slice grew since last month". Growth concentrated in one customer is a product problem; growth spread evenly is a traffic problem; and the two need completely different responses.

Tracking checklist

• Usage is read from every response and stored with your own request ID.
• The feature label is attached at the call site.
• Raw token counts are stored, not only derived credits.
• Cache counters are captured where caching applies.
• Retries are recorded as separate attempts.
• Daily spend against a budget threshold is alerted on.
• Tokens per successful request is tracked, not only total volume.
• A day's totals are reconciled against the platform console.
• Rates are read from the current catalog rather than hard-coded.

Further reading

• GPT Model Selection Guide — the tier decision that drives most of your per-request cost.
• Prompt Caching Guide — reducing the input cost that dominates chat workloads.
• Production Readiness Checklist — where usage tracking belongs in a launch checklist.

FAQ

Related reading

• Qwen Cost Optimization
• Token Usage and Cost Tracking
• Embedding Model Selection Guide