What Is an AI API Gateway? How a Relay Works, Hop by Hop

An AI API gateway is a service that accepts an API request in a standard format, resolves it to a specific model on a specific upstream provider, forwards it, and returns the response in the same standard format. The client only ever talks to the gateway; the provider credentials, the routing decision and the model naming live on the gateway side.

That definition is deliberately mechanical, because most confusion about gateways comes from mixing up three different things: protocol translation, credential management, and routing. This article walks through one request hop by hop, then shows what the relay can and cannot change about your results.

One request, six hops

Take a plain chat completion sent to https://exaiv.com/api/ai/v1/chat/completions. The request does not travel in one step; it passes through distinguishable stages, and each one can fail in a different way.

Two of those hops explain almost every support question. Hop 3 is why a model name that works elsewhere can return a not-found error here: the catalog on this side is the authority, not the name you memorised from another provider. Hop 4 is the security-relevant one: your key identifies you to the gateway, and the upstream provider never sees it.

What the gateway does — and what it does not

The last row of the right column is the one teams forget. A gateway makes calls uniform; it does not make model output correct. Parsing, schema validation and retry policy remain your code's job.

Why a relay exists at all

If a gateway is just a middle hop, why not call the provider directly? Three practical reasons come up repeatedly, and none of them is about hiding anything:

One key, many providers. A product that uses a fast model for classification and a larger one for reasoning would otherwise need two accounts, two billing relationships and two sets of credentials in production. With a relay, the application holds one credential.

A stable contract in front of a changing catalog. Providers rename models, publish dated snapshots and retire versions. If the model id appears in one place on the gateway side, an application can absorb part of that churn by changing configuration instead of code.

A single egress path for regional clients. For developers in Southeast Asia or the EU, the network path to a given provider is an operational variable: it can be fast on one carrier and slow on another. A relay offers one endpoint with a known behaviour instead of several provider endpoints with unknown ones. It does not remove latency; it makes the endpoint you integrate against a constant.

The part most people miss: the protocol is the same locally

The strongest argument for learning the OpenAI-compatible request shape is not that a relay uses it. It is that almost every serious local inference server uses it too. Ollama, llama.cpp's server and vLLM all expose an OpenAI-compatible endpoint, so the same client library, the same message array and the same streaming parser work against a model running on a laptop and against a relay.

That means a sane architecture has exactly one variable for the model location:

import os
from openai import OpenAI

# Two targets, one code path: only base_url and model change
TARGETS = {
    "relay": {
        "base_url": "https://exaiv.com/api/ai/v1",
        "api_key": os.environ["EXAIV_API_KEY"],
        "model": "deepseek-v4.1-flash",
    },
    "local": {
        "base_url": "http://localhost:11434/v1",   # Ollama default OpenAI-compatible endpoint
        "api_key": "ollama",                        # local servers usually ignore the key value
        "model": "qwen3.8-flash",
    },
}

def ask(target: str, prompt: str) -> str:
    cfg = TARGETS[target]
    client = OpenAI(base_url=cfg["base_url"], api_key=cfg["api_key"])
    r = client.chat.completions.create(
        model=cfg["model"],
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return r.choices[0].message.content

print(ask("relay", "Summarise this changelog in three bullets."))

The same idea with curl, useful for checking a relay before any SDK is involved:

curl -s https://exaiv.com/api/ai/v1/chat/completions \\
  -H "Authorization: Bearer $EXAIV_API_KEY" \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "kimi-k3",
    "messages": [{"role": "user", "content": "Reply with the single word: ok"}],
    "temperature": 0
  }'

A 401 here means the key is missing or wrong; a 404 mentioning the model means the identifier does not exist in the catalog. Those two errors account for most first-attempt failures, and they are distinguishable without reading any documentation.

When a relay is the wrong tool

A gateway is an architectural choice with trade-offs, and there are cases where calling a provider directly is correct:

• Data residency requirements that name the provider. If a compliance rule specifies who may process the data, an intermediary is an additional party in that analysis. Check the requirement before assuming a relay is compatible with it.
• A single provider and a latency budget measured in single-digit milliseconds. Every additional hop is an additional place for time to be spent. If the workload is latency-bound and uses one provider, direct access is simpler to reason about.
• Air-gapped environments. If the network cannot reach the public internet, the only endpoint available is the one running next to the model. This is exactly where the local OpenAI-compatible server is the whole answer.

None of these make a relay bad; they make it a fit question rather than a default.

Troubleshooting by hop

The pattern to internalise: status codes tell you which hop failed. A 401 is never a model problem, and a model-not-found is never an authentication problem. Diagnostics get much faster when you stop treating the gateway as one opaque box.

How to evaluate a relay in ten minutes

Documentation describes intent; these five tests describe behaviour. Run them before you commit a codebase to an endpoint.

One, send a minimal request and keep the raw response. Compare the field names it returns with the ones your code reads. A first request that returns 200 but omits a field you depend on is a design problem, not a bug you will fix later.

Two, send an intentionally invalid model identifier. Read the error body. An endpoint that names the unknown model is easier to operate than one that returns a generic failure, because your retry logic can distinguish a configuration mistake from a transient fault.

Three, hold a streaming request open for longer than any proxy timeout in your path. Corporate proxies and some CDNs terminate idle connections; a stream that stalls after sixty seconds works on a laptop and fails in the office. This test also tells you whether the client needs an explicit read timeout.

Four, push a request past the context limit on purpose. The refusal should appear as a clear error before any tokens are billed, not as a truncated answer you notice in production.

Five, call the same model twice with the same input and temperature set to zero. The outputs will not be byte-identical, and that is expected. What you are checking is that the response shape is stable: same keys, same types, same nesting. Shape drift is the failure mode that breaks downstream code silently.

A short checklist before you adopt one

• I know which catalog the model identifiers must come from, and where to read it
• I know which parts of the request my application validates, and which parts the gateway normalises
• I have a fallback plan for a model id that disappears from the catalog
• I have tested streaming through my own network, not only from a laptop on a clean connection
• I have decided whether the workload needs a relay, a direct provider, or a local server
• I have verified that the same client code runs against both the relay and a local endpoint

Related reading

• Chat Completions reference
• Model catalog
• Pricing and credits
• Local deployment playbook
• Kimi K3 API guide