Batch Inference in Practice: Running Offline Workloads Faster and Cheaper

LLM services carry two kinds of load: latency-sensitive online requests like chat, and offline tasks like "classify ten thousand documents" where nobody waits for a single result and only total time matters. Calling the latter one at a time through an online endpoint is both slow and wasteful. Batch inference organizes these workloads into throughput-first pipelines — this guide shows how.

Throughput and latency are different goals

The test is simple: is a user waiting for this one result? If yes, online; if no, it belongs in a batch queue.

Why continuous batching works

Traditional batching holds resources until the entire batch finishes, so one slow request stalls everyone. Continuous batching lets a finished request release its slot immediately to a queued one, keeping the GPU nearly idle-free. Frameworks like vLLM and SGLang implement this.

For the caller, the lesson is: feeding the server steadily beats "send one huge batch and wait". Keep the queue stocked so the server can keep batches full.

Organizing an offline batch job

1. Slice inputs and deduplicate. Split the job into independent units; deduplicate identical inputs so they are not computed twice.

2. Fix parameters and prompts. One temperature, max_tokens, and system prompt across the batch for comparable results.

3. Concurrency control plus rate-limit retry. Use bounded concurrency (for example 4-8), exponential backoff on 429/503, and record failed units individually instead of aborting the whole batch.

import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://exaiv.com/api/ai/v1")

def run_one(item, retries=4):
    for attempt in range(retries):
        try:
            return client.chat.completions.create(
                model="kimi-k3",  # per the live /models catalog
                messages=[{"role": "user", "content": item}],
                temperature=0,
            ).choices[0].message.content
        except Exception:
            if attempt == retries - 1:
                raise
            time.sleep(2 ** attempt)

results, failures = {}, []
with ThreadPoolExecutor(max_workers=6) as pool:
    futures = {pool.submit(run_one, it): i for i, it in enumerate(items)}
    for fut in as_completed(futures):
        i = futures[fut]
        try:
            results[i] = fut.result()
        except Exception as e:
            failures.append((i, str(e)))

4. Persist results and support resume. Write each result with its input id; after an interruption, skip completed ids instead of restarting from scratch.

Common batch-job pitfalls

Overlong context: split long documents first; do not pass a whole book (see the context engineering guide).

Output parse failures: for structured tasks, require JSON output and retry failed units individually without polluting the batch.

Duplicate billing: on retry, confirm you did not produce duplicate successful results; use input ids as idempotency keys.

Silent quality drift: batch jobs are especially prone to "finished but changed behavior". Spot-check 5-10% of outputs manually, or run a fixed eval set as regression (see the LLM evaluation guide).

One complete example: classifying ten thousand documents

Putting the flow together: assign 10000 documents to 8 categories.

1. Preprocess: truncate each document to 2000 characters, deduplicate by content hash, giving 9400 independent units.

2. Fix configuration: temperature 0, JSON structured output, one identical system prompt.

3. Run at concurrency 6: use the skeleton above, persisting each result with its input id.

4. Handle failures: retry 429/503 units individually; re-run parse-failed units once without polluting the batch. A single malformed output should never force a full restart.

5. Resume: if the network drops, restart and skip ids already persisted. This is why idempotency keys matter as much as retry logic — without them, a retry can silently duplicate both work and cost. Track a per-unit status (pending, done, failed) rather than relying on in-memory counters, because a long batch job will outlive at least one process restart in practice. Also record elapsed time per unit; a rising tail latency is often the first sign that concurrency is too high or context is too long.

6. Quality spot-check: randomly sample 500 results (about 5%) for manual category verification, and run one regression on a fixed eval set.

The most common mistake is opening 64 concurrent requests on the first attempt: the cloud returns continuous 429s and a local server runs out of VRAM from KV cache. Start at 4-8 and tune from feedback — that is how stable throughput is built. Keep the numbers from each run (items per minute, failure rate, retry count, average context length). They turn batch tuning from guesswork into a repeatable engineering practice, and they make the next job's configuration a one-line decision instead of a fresh experiment.

What to measure in production

Once a batch job runs regularly, four numbers tell you whether it is healthy: items completed per minute, failure/retry rate, average context length per item, and tail latency (the slowest few percent of units). A rising retry rate usually means concurrency exceeded the rate limit or the server's KV cache budget. A rising average context length means document slicing regressed. Tail latency climbing while throughput holds steady often signals a few oversized units monopolizing slots — cap input size per unit.

Keep these numbers next to the eval-set regression results: a batch job can be fast and cheap while quietly producing worse outputs, and only the eval catches that. Also archive the configuration (model ID, parameters, concurrency, prompt version) with each run so a later quality change can be attributed to a specific change rather than guessed at. This is the same discipline as model version management, applied to offline workloads.

Online versus offline: a decision table

Most products have both classes of work. A support chatbot is online; nightly re-embedding of new documents is batch. Keep them on separate paths with separate monitoring, because optimizing one for the other's metric hurts both: batching an interactive path adds latency, and giving an offline job priority over chat wastes GPU on work nobody is waiting for. Sharing one OpenAI-compatible endpoint keeps the application code simple, but the operational policies should differ: online paths get tight timeouts and immediate errors, batch paths get generous retries and idempotent writes. Document which class each workload belongs to, and the concurrency, timeout, and retry settings follow from that classification instead of being tuned ad hoc.

Failure taxonomy for batch jobs

Not every failure deserves the same response. Classify them before writing retry logic:

• Transient (429, 503, network timeout): retry with exponential backoff; these resolve themselves.
• Permanent (401, invalid model ID, malformed request): do not retry — fix configuration and re-run only affected units. A retry loop here just burns quota.
• Parse failures (invalid JSON, truncated output): re-run the unit once with a stricter format instruction; if it fails again, quarantine it for inspection.
• Silent quality failures (valid output, wrong content): only a fixed eval set or spot-check catches these. They are the reason batch jobs need evaluation, not just monitoring. Keep the eval results with the run record so a regression can be attributed to a specific configuration. A batch job that reports 100% success may still be producing unusable data — success means "the request completed", not "the answer was right". Distinguish operational success from task success, and measure both.

FAQ

Related reading

• RAG Knowledge Base Guide — retrieval and context together
• LLM Evaluation Methodology — regression with a fixed eval set
• Qwen Cost Optimization — controlling token consumption
• VRAM Calculation in Practice — concurrency VRAM ceiling
• Pricing — live cloud pricing