Rate Limits and Concurrency on EXAIV: Controlling Request Rate Without Slowing Down

Throughput problems have two opposite failure modes. Send too few requests in parallel and your application is slower than it needs to be, wasting a fast model on a queue. Send too many and you hit a limit, and the result is worse than slow: retries, backoffs, and eventually failures that reach the user. Most systems that suffer from the second problem did not set a limit at all — they simply let every incoming request open its own upstream call.

The fix is to make concurrency a deliberate decision: a bounded pool, a rate limit you control, and a clear reaction when the platform's own limit is reached.

Two different limits

The terms get used interchangeably, but they describe different constraints:

A client can respect the rate limit and still exceed the concurrency limit — ten requests per second, each taking thirty seconds, means three hundred requests in flight. Conversely, a client can be well under the rate limit and still exceed the concurrency limit with a burst.

Check the platform's current documentation for the specific limits that apply to your account. The numbers change; the structure does not.

Sizing a concurrency pool

The pool size should follow from two facts: how fast you want the queue to drain, and how long a single request takes.

import asyncio, random

MAX_CONCURRENCY = 8
client = OpenAI(
    api_key="YOUR_EXAIV_API_KEY",
    base_url="https://exaiv.com/api/ai/v1",
)

semaphore = asyncio.Semaphore(MAX_CONCURRENCY)

async def one_call(messages):
    async with semaphore:                      # never more than 8 in flight
        response = await client.chat.completions.create(
            model="gpt-6-sol",
            messages=messages,
            max_tokens=1024,
        )
        return response.choices[0].message.content

The same idea in Node.js, using a simple counting wrapper:

let inFlight = 0;
const waiters = [];

async function acquire() {
  if (inFlight < MAX_CONCURRENCY) { inFlight++; return; }
  await new Promise((resolve) => waiters.push(resolve));
  inFlight++;
}

function release() {
  inFlight--;
  const next = waiters.shift();
  if (next) next();
}

Three rules for a pool that behaves well:

• Bound the queue as well as the pool. If you accept unlimited work into a queue, a traffic spike becomes unbounded latency. Reject or shed load past a threshold.
• Make the timeout longer than the pool wait. Otherwise queued requests time out before they are ever sent.
• Prefer a smaller pool with faster models for bulk work. Concurrency is often a blunt instrument; a cheaper tier per request reduces the pressure on the pool as well.

Smoothing bursts with a token bucket

Rate limits are usually defined over a window — requests per second or per minute. A burst that exceeds the average will trip the limit even if the long-run average is fine. A token bucket smooths exactly this: tokens are added at a steady rate, and each request consumes one.

import asyncio, time

class TokenBucket:
    def __init__(self, rate_per_sec, burst):
        self.rate = rate_per_sec
        self.burst = burst
        self.tokens = burst
        self.updated = time.monotonic()
        self.lock = asyncio.Lock()

    async def acquire(self):
        while True:
            async with self.lock:
                now = time.monotonic()
                self.tokens = min(self.burst, self.tokens + (now - self.updated) * self.rate)
                self.updated = now
                if self.tokens >= 1:
                    self.tokens -= 1
                    return
                wait = (1 - self.tokens) / self.rate
            await asyncio.sleep(wait)

Use it in front of the semaphore: the bucket controls the arrival rate, the semaphore controls how many run at once. Both are needed; neither replaces the other.

For a serverless function with no long-lived process, the equivalent is a shared external counter — for example a Redis token bucket — because each invocation has its own memory.

Reacting to a 429

When a 429 arrives, the response is telling you to send less. A sound reaction:

1. Read Retry-After if the header is present, and wait at least that long.
2. Back off exponentially with jitter if the header is absent.
3. Shrink the effective pool for a period. A simple approach is to halve the concurrency after each 429 and restore it slowly as successful requests accumulate.
4. Stop queueing new work while the limit is being respected, if the backlog is growing.

That last point is the difference between recovering in seconds and recovering in minutes. A queue that keeps accepting work during a limit event grows the backlog, and every item in it is a request that will fail or take a long time.

Finding your ceiling

Measure rather than guess. A simple approach:

1. Start with a small pool, such as four concurrent requests.
2. Send a steady stream of representative requests for a minute.
3. Record success rate, p95 latency, and 429 count.
4. Increase the pool and repeat until you see 429s appear or latency degrade sharply.
5. Settle at roughly the point before the degradation starts, with headroom for traffic spikes.

The result is a number with evidence behind it. Note that it is specific to the model, the request length, and the time of day — re-measure after changing any of those.

Queueing, backpressure, and the user experience

Whatever you do, decide explicitly what happens to a request that cannot start immediately:

For an interactive product, a fast, honest failure that the user can retry is usually better than a request that silently waits two minutes. For a background job, a bounded queue with a generous timeout is the right default.

Rate limits in serverless environments

A serverless function has no long-lived memory, which means the concurrency pool and the token bucket you wrote for a traditional server do not exist by default. Each invocation is independent, and each one can open its own upstream request.

That turns an ordinary rate limit into a different problem: not "how many requests are in flight" but "how many invocations happen at once". When a traffic spike arrives, the platform starts hundreds of instances, each of which calls the model, and the upstream sees a burst that no client-side pool was designed to absorb.

Ways to handle it:

A shared limiter. Keep the counter outside the function — a rate-limited API, a Redis token bucket, or a platform feature that limits concurrency across instances. This is the only approach that actually bounds the burst.

A limit at the edge. Rate limiting in front of the function, before it is invoked, bounds the request rate. An API gateway or a WAF rule that rejects excess traffic costs nothing downstream, because the function never starts.

A lower per-instance limit. Each function caps its own concurrent upstream calls at one or two. This does not bound the total, but it stops a single instance from fanning out.

Fewer, longer invocations. If the workload allows it, one invocation that processes a batch of items uses far less overhead than many invocations that each process one.

Rate limit checklist

• Upstream concurrency is capped by an explicit pool, not by whatever the traffic happens to be.
• The queue is bounded, and overflow behavior is defined.
• The request rate is smoothed with a token bucket or equivalent.
• 429 responses trigger backoff and a reduction in effective concurrency.
• Retry-After is respected when present.
• The throughput ceiling has been measured, and the pool is set below it.
• Queue depth, wait time, and pool utilization are monitored.
• Retries are counted separately from successful requests, because they add load.

Further reading

• Error Handling and Retry Guide — what to do once a limit has been hit.
• Token Usage and Cost Tracking — the metrics that show whether concurrency is buying throughput or just adding load.
• Production Readiness Checklist — where rate limiting belongs in a launch review.

FAQ

Related reading

• Error Handling and Retry Guide
• Qwen Cost Optimization
• Batch Inference Guide

Terminology and pre-flight checklist

Total parameters, active parameters and the KV cache are three different quantities. Total parameters are all the weights that must be loaded into memory; active parameters are the subset that computes a single token. KV cache is temporary memory that grows linearly with context length, so a 1M window is a ceiling rather than a free default. Quantization lowers the precision of stored weights to shrink files; it does not reduce the parameter count. A vision_config in a config.json describes architecture, not a wire format you may safely invent.

Before shipping anything built on these notes, run six checks. Re-read the numbers from the live source on the day you publish, because download counters move weekly. Open the live /models catalog and confirm the model ID you plan to call is still there. Open /pricing rather than copying a price into your application. Never hard-code a request body you have not validated against the official reference. Record the model ID, input and output token counts, and latency for a representative set of prompts, then keep that set unchanged so a later model swap produces a comparable reading. Read the licence on the model card before commercial use, since vendor-specific terms are not a permissive default. Add exponential backoff for retryable requests, set a timeout, and log the error class so that a 401, a 404 and a 429 never get the same treatment.