Production Readiness Checklist for EXAIV: Before You Point Real Traffic at the Gateway

An integration that works in development and fails in production usually fails for one of a small number of reasons: the timeout was never set, the retry loop has no cap, nobody measured the cost, or the key was hard-coded somewhere convenient. Each of those is cheap to fix before launch and expensive to fix after.

This checklist is the review to run before real traffic starts. It is organized as a sequence of questions with the specific thing to verify, rather than as generic advice. Most items take minutes to confirm, and each one is a failure you will otherwise meet under load.

1. Credentials

• The key is stored in the deployment platform's secret store, not in the repository or an image.
• The application reads it from the environment at start-up and fails fast with a clear message if it is missing.
• Development, staging, and production use different keys.
• The key can be revoked and replaced without a rebuild.
• No key appears in logs, error messages, screenshots, or shared documents.
• A revocation drill has been performed: a revoked key produces a recognizable error.

Details in the API key management guide.

2. Model selection

• The model ID was copied from the live catalog, not from memory or an old ticket.
• The chosen tier was compared against a cheaper one on a real evaluation set.
• The evaluation set is stored in version control together with the result.
• The model ID is configured rather than hard-coded in business logic.
• There is a plan for what happens when a model is retired or renamed.

See the GPT model selection guide.

3. Timeouts

• A connect timeout is set.
• A read timeout is set, and it is longer than the slowest acceptable generation.
• The total request deadline is shorter than the caller's own timeout, so your application fails before the caller does.
• Streaming requests have their own timeout, appropriate to a long-lived connection.
• A timeout does not trigger an unbounded retry.

4. Retries and error handling

• Only 429 and 5xx responses are retried; 400, 401, 403, and 404 fail fast.
• Backoff is exponential with jitter, and Retry-After is respected.
• Both the attempt count and the total elapsed time are capped.
• The error body is logged with a request ID, and the authorization header never is.
• Every failure class has a defined user-facing behavior.
• Retries are counted as requests, so retry cost is visible in usage reporting.

Details in the error handling and retry guide.

5. Concurrency and rate limits

• Upstream concurrency is capped by an explicit pool.
• The queue is bounded, and the overflow behavior is defined.
• Request rate is smoothed with a token bucket or equivalent.
• A 429 response reduces load rather than increasing it.
• The throughput ceiling was measured under representative traffic.

See the rate limits and concurrency guide.

6. Cost control

• Credits per request are estimated from measured token counts, and the estimate was checked against a real invoice period.
• A daily or monthly spend alert exists.
• A hard ceiling exists that stops or degrades traffic rather than only notifying.
• Output length is bounded, and the max_tokens values are set per use case rather than to a global maximum.
• A token or output cap exists for any endpoint that accepts arbitrary user input.
• The most expensive model in the catalog is not reachable from a low-trust code path.

7. Observability

• Every request records: request ID, feature, model, input tokens, output tokens, cached tokens, latency, and final status.
• The usage record is joined to the application's own logs by request ID.
• Errors are counted by type, and retries are counted separately from successes.
• Queue depth and pool utilization are visible.
• There is a dashboard a non-author can read.
• Usage data is retained long enough to compare periods.

See the token usage and cost tracking guide.

8. Output correctness

• finish_reason is checked, and length is treated as truncation.
• Outputs consumed by code are validated against a schema before use.
• Tool arguments are validated and authorized before execution.
• Structured output requests specify a schema rather than asking for JSON in prose.
• The truncation and malformed-output paths have been tested with a deliberately truncated response.

9. Prompt structure and caching

• Stable content comes first and volatile content last in the prompt.
• No timestamp, request ID, or other changing value sits in the system prompt.
• The cache hit ratio is recorded, so caching effectiveness is visible.
• Long prompts were shortened, not only reordered.

See the prompt caching guide.

10. Data handling and security

• The review covered which party receives prompts, how they are logged and retained, and how keys are stored.
• Personally identifiable information is handled according to your own policy, and the policy is written down.
• Credentials, secrets, and sensitive files are filtered out of prompts.
• The application does not log full prompt or completion content unless that is deliberate and reviewed.
• Rate limiting and authentication exist on the application's own endpoints, not only on the gateway.
• A key that may have been exposed has been rotated.

11. Operations

• A rollback path exists: the previous model ID, prompt version, or configuration is retained.
• The deployment can be reverted without a data migration.
• Someone is responsible for the alerts, and that is written down.
• A model catalog refresh is checked on a schedule, not only when a request fails.
• The incident path is documented: what to check, what to try, when to escalate.

12. Failure drills

This is the section that proves the other eleven. Run each drill against a non-production environment, and confirm that the application behaves as intended rather than as hoped.

The five questions that catch most problems

If time is short, these five cover the largest share of launch incidents:

1. What happens when the key is wrong? If the answer is a hang or an infinite retry, fix it before launch.
2. What is the maximum cost of one request? If the answer is "whatever the model produces", you need a cap.
3. What happens when the response is truncated? If the answer is "it looks like an answer", you need a check.
4. How many requests can run at once, and who decided that number? If there is no answer, you have an unbounded pool.
5. Can you turn this feature off? If the only switch is a redeploy, you do not have a rollback path.

Launch review checklist

• Every section above was verified by testing, and the evidence is recorded.
• The model IDs, parameters, and prompt versions in use are written down with the reasons for them.
• All eight failure drills were run and behaved as expected.
• Alerts are live and reach a person, not just a dashboard.
• A rollback was tested, not just documented.
• Someone other than the author reviewed the result.
• The first week after launch has a defined check-in, with the metrics to compare.

Further reading

• GPT Model Selection Guide — the tier decision behind most of the cost and quality outcomes.
• Error Handling and Retry Guide — the retry behavior the drills verify.
• Token Usage and Cost Tracking — the observability the review depends on.

FAQ

Related reading

• Error Handling and Retry Guide
• Model Version Management Guide
• Local Compute Cost Model