Error Handling and Retry Strategy for EXAIV: Turning Failures into Controlled Behavior

An API client that fails cleanly is more valuable than one that almost never fails. The gateway can be healthy while your request is malformed, your key is wrong, your account is out of credit, or the upstream provider is briefly saturated. Each of those failures wants a different response, and treating them all the same — retry everything, or fail everything — turns a recoverable blip into an outage or a short outage into a long one.

This guide covers how to read an error response, how to classify failures by what you should do about them, how to retry safely with backoff and jitter, and how to keep a retry storm from turning one slow upstream into a self-inflicted denial of service.

What an error response contains

Errors from the endpoint return a JSON body with a message and, usually, a type and a request identifier:

{
  "error": {
    "message": "Rate limit reached for requests per minute",
    "type": "rate_limit_error",
    "param": null,
    "code": null
  }
}

Log the whole object, plus the HTTP status, plus your own request ID. The message is the fastest way to tell "your key is wrong" from "that model does not exist" from "the account is out of credit" — three problems with three completely different fixes.

Classifying failures

The distinction that matters most is between "this request is wrong" and "the service is temporarily unable". Retrying a wrong request only wastes time and money — and in the case of a malformed request that is expensive to generate, it wastes credits too.

The error codes on the Claude-compatible endpoint are covered in detail in the Claude API error codes guide.

Deciding whether a failure is worth retrying

Before adding a retry, answer three questions:

1. Would the identical request succeed later? A 400 will not. A 503 might.
2. Is the operation safe to repeat? Chat completions are read-only in effect — repeating them is safe. An operation that creates state, charges a card, or sends a message may not be, and for those you need an idempotency key or a dedupe record before retrying.
3. What does the retry cost? A retry consumes a request and, if it succeeds, produces a full response. Two retries on a 30-second generation is three times the work.

Exponential backoff with jitter

The correct shape: wait longer after each failure, and add randomness so that many clients do not all retry at the same instant.

import random, time

def with_retry(fn, *, attempts=4, base_delay=1.0, max_delay=30.0):
    last_error = None
    for attempt in range(attempts):
        try:
            return fn()
        except Exception as exc:
            last_error = exc
            status = getattr(exc, "status_code", None)

            # Do not retry a request that will fail identically.
            if status is not None and status < 500 and status != 429:
                raise

            if attempt == attempts - 1:
                break

            delay = min(base_delay * (2 ** attempt), max_delay)
            delay = random.uniform(0, delay)      # full jitter
            time.sleep(delay)

    raise last_error

Why the jitter matters: without it, every client that failed at the same moment computes the same next retry time and hits the service in a synchronized wave. The wave keeps the service overloaded and produces the second failure. Randomizing the delay spreads the retries out.

Retry-After, when the server sends it, beats any delay you compute yourself. Read the header and use it.

Preventing a retry storm

A retry storm is what happens when every client retries at once, or when a queue keeps pulling new work while the upstream is still failing. Three defenses:

Cap the attempts. Four attempts with exponential backoff is a reasonable default. Beyond that, the failure is not transient and the caller should handle it.

Cap the total elapsed time. A deadline is more meaningful than an attempt count. If the caller needs an answer within five seconds, three slow retries are worse than one fast failure.

Bound the queue. If a worker pool is pulling from a queue, stop pulling while the upstream is failing. Otherwise each worker is busy failing and retrying while the queue grows, and recovery takes much longer than it should.

Handling a 429 specifically

A rate-limit response is a clear instruction: you are sending too much, too quickly. Three useful responses:

1. Read Retry-After and wait at least that long. It is often the fastest correct answer.
2. Reduce concurrency. If your client opens many parallel requests, the limit will keep being hit no matter how long you wait between attempts. Lower the pool size.
3. Add client-side pacing. A token bucket in front of the client smooths bursts into a steady rate, which is what a limit actually asks for.

Also check that the response is not a quota exhaustion rather than a short-term rate limit — the message usually distinguishes them, and the remedies differ. Quota exhaustion does not go away by waiting a second.

Timeouts

Set both a connection timeout and a read timeout, and make the read timeout comfortably longer than the slowest acceptable generation. A timeout that fires while the model is still producing produces a wasted request: the work happened, the answer was discarded, and the retry repeats it.

If your client has a global timeout shorter than a long generation, either lower max_tokens for that path or use streaming so that progress is visible and the connection is not idle.

What the user should see

Not every failure belongs in front of a user. A sensible ladder:

Idempotency for operations that are not

Chat completions are effectively read-only: repeating the same request produces another answer of the same kind, and a retry is harmless. Any operation with a side effect is different. Sending a message, creating a record, charging a card, or triggering a workflow twice is not harmless, and a naive retry policy will do exactly that.

Three ways to keep a retry safe:

An idempotency key. Generate a unique identifier for the logical operation, send it with the request, and have the server deduplicate on it. A repeated request with the same key returns the original result instead of performing the work again.

A dedupe record. If the server cannot be changed, record the operation key and its result in your own store before retrying, and check that store before sending. This is more work than it sounds, and it is still better than a duplicate charge.

A state machine. Model the operation as a sequence with explicit states, and retry only the transition that is safe to repeat. "Send, then confirm" is not one step; it is two, and only the second is safe to repeat.

The rule to apply to any retry you write: know what happens if the request is processed twice. If the answer is "the user receives two messages", the retry policy needs one of the three mechanisms above before it ships.

A retry checklist

• Error responses are parsed and the body is logged with a request ID.
• The Authorization header is never included in logs.
• Only 429 and 5xx are retried; 400, 401, 403, and 404 fail fast.
• Retry-After is respected when present.
• Backoff is exponential with jitter.
• Both the attempt count and the total elapsed time are capped.
• The upstream is not hammered while it is failing — the queue is paused or a circuit breaker is open.
• Every retry attempt is counted and observable.
• The user-facing behavior for each failure class is defined.

Further reading

• Rate Limits and Concurrency Guide — reducing the request rate that causes 429 responses in the first place.
• Streaming Responses Guide — handling a stream that ends early, which is a failure mode retries cannot fix.
• API Key Management and Rotation — the response path for a 401.

FAQ

Related reading

• Qwen API Troubleshooting
• Error Codes Reference
• Production Readiness Checklist

Terminology and pre-flight checklist

Total parameters, active parameters and the KV cache are three different quantities. Total parameters are all the weights that must be loaded into memory; active parameters are the subset that computes a single token. KV cache is temporary memory that grows linearly with context length, so a 1M window is a ceiling rather than a free default. Quantization lowers the precision of stored weights to shrink files; it does not reduce the parameter count. A vision_config in a config.json describes architecture, not a wire format you may safely invent.

Before shipping anything built on these notes, run six checks. Re-read the numbers from the live source on the day you publish, because download counters move weekly. Open the live /models catalog and confirm the model ID you plan to call is still there. Open /pricing rather than copying a price into your application. Never hard-code a request body you have not validated against the official reference. Record the model ID, input and output token counts, and latency for a representative set of prompts, then keep that set unchanged so a later model swap produces a comparable reading. Read the licence on the model card before commercial use, since vendor-specific terms are not a permissive default. Add exponential backoff for retryable requests, set a timeout, and log the error class so that a 401, a 404 and a 429 never get the same treatment.