Choosing an LLM Gateway: Seven Things That Actually Matter

Most teams adopt an LLM gateway for one of two reasons: they got paged when a single provider went down, or finance asked why the API bill doubled and nobody could answer. A gateway sits between your application and the model providers, so it inherits every reliability and correctness requirement your application has. That makes the selection decision more consequential than it looks.

Feature matrices for gateways all look the same. Here are the seven things we have found actually separate a gateway you can run in production from one you will rip out in six months.

1. API compatibility

The gateway's API surface determines your migration cost. If it speaks the OpenAI chat completions format, you point your existing SDK at a new base URL, swap the key, and you are done — zero client code changes:

curl https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4",
    "stream": true,
    "messages": [{"role": "user", "content": "Summarize this ticket."}]
  }'

The word "compatible" hides a lot, though. Test the edges, not the happy path. Streaming is where compatibility claims break first: verify that SSE chunk framing, finish_reason values, and the final usage block match what your client library expects. Tool calling is the second failure point — check that parallel tool calls, streamed tool-call argument deltas, and tool_choice options survive translation across providers. A gateway that silently drops a tool-call delta will corrupt your agent loop in ways that are miserable to debug.

2. Routing and failover

Failover is the headline feature, so ask exactly how it works. Static retry-on-error is table stakes. What you want is health-based routing: the gateway should track per-provider error rates and latency, stop sending traffic to a degraded upstream before your users notice, and recover automatically when the upstream does.

Two questions expose weak implementations. First: what happens when a provider fails mid-stream? Retrying a request that already emitted tokens to the client is not straightforward — a good gateway either restarts the request on a secondary provider before any tokens are flushed, or clearly documents that mid-stream failures surface as errors. Vague answers here mean untested code. Second: can you define fallback chains per model, so that an outage on your primary provider degrades to an equivalent model rather than a hard failure?

3. Rate limiting and quotas

Once multiple services share one gateway, one runaway batch job can starve your production traffic. You need per-key limits on both requests per minute (RPM) and tokens per minute (TPM) — token limits matter more, since a single request can carry a 100k-token context.

Look for fair queuing rather than plain rejection. When aggregate demand exceeds upstream capacity, the gateway should queue and interleave requests across keys instead of letting whichever client retries most aggressively win. Also check that limit changes take effect without a deploy; you will be adjusting these weekly at first.

4. Cost tracking

LLM spend is unusual: it is highly variable, driven by prompt engineering decisions made far from the billing owner, and often split across providers with different pricing. The gateway is the only component that sees every request, so it is the natural place to answer "who spent what."

The minimum bar is token-level attribution: every request logged with input tokens, output tokens, cached tokens (if the provider discounts them), the model used, and the key or team that made the call. On top of that you want budgets — hard or soft caps per key or per project — and alerts that fire at thresholds like 80% of budget, not after the money is gone. If the gateway can only tell you total spend per day, you will be exporting logs to a spreadsheet within a month.

5. Observability

When a user reports "the AI feature is slow," you need to answer three questions fast: which provider, which model, and is it us or them. That requires request-level logs with timing breakdowns (queue time, upstream time to first token, total duration), latency percentiles per model and provider — p50 tells you nothing about tail pain, so insist on p95 and p99 — and a real error taxonomy.

Error taxonomy deserves emphasis. "HTTP 500" is not a diagnosis. A useful gateway distinguishes upstream rate limits, upstream timeouts, content filter refusals, malformed requests, and its own internal errors, because each has a different fix. Bonus points for exportable traces so you can join gateway data with your existing APM.

6. Security

A gateway holds your provider API keys and sees every prompt, which may include customer data. Evaluate it like any other component in your data path. Provider keys should be stored encrypted, never returned by any API, and rotatable without downtime. Traffic must be TLS in transit and any logged payloads encrypted at rest.

Then read the data retention policy, not the marketing page. How long are request and response bodies kept? Can you turn body logging off entirely while keeping metadata for billing? Is there a written commitment that neither the gateway vendor nor its subprocessors train models on your traffic? If you handle regulated data, ask for the compliance documentation before the pilot, not after.

7. Operational burden

Finally, be honest about who runs this thing. Self-hosting an open-source gateway gives you full control and keeps traffic inside your VPC, but you now own upgrades, provider API changes (they change often), scaling the gateway itself, and its uptime — the gateway is a single point of failure by design, so it must be more reliable than any provider behind it.

A managed gateway inverts the trade: you give up some control and add a network hop, and in exchange the vendor carries on-call, upstream API churn, and capacity. For most teams under, say, ten engineers, the managed option is cheaper once you price in the on-call load honestly. Whichever you choose, demand a published uptime target with history behind it — 99.9% or better — and a status page you can subscribe to.

The checklist

Condensed into a table you can take into vendor calls:

Nothing on this list is unusual — it is the same bar you would set for any load balancer or API proxy in your critical path. The teams that get burned are the ones that evaluated the gateway as an AI product instead of as infrastructure. We build the Runix gateway against this exact list; if you want to compare notes on your evaluation, get in touch.