How to build an LLM gateway for cost, quality, and resilience

When an AI bill starts rising faster than the team using it, the first instinct is usually to look for a cheaper model. While that can help, in most organizations, the bigger savings sit in the system around the model: the context each request carries, the loops an agent runs, what gets cached, and which model handles which task.

Managing that system gets much easier when every model call goes through one layer your team controls. That layer, known as an LLM gateway, becomes the core of your AI infrastructure. It lets teams test a cheaper model, add a new provider, or tighten a policy without touching each individual application. 

What is an LLM gateway?

An LLM gateway, sometimes called an AI gateway, is a middleware layer between your applications and the model providers they call. Instead of each application keeping a direct API connection to OpenAI, Anthropic, Google, or a self-hosted model, every request goes through the gateway. The LLM gateway acts as a single integration point and applies the same rules every time.

It works much like an API gateway does for web services, but with features built for AI models. Many gateways expose an OpenAI-compatible API, giving developers a unified interface to multiple LLM providers without changing app code. Switching models or providers then becomes a configuration change rather than a rewrite of application code.

A well-designed gateway handles three jobs:

Why an LLM gateway matters now

It controls the input tokens that drive AI costs

AI coding is unusually input-heavy. In a study of roughly 100 trillion tokens of real-world LLM traffic, programming prompts averaged three to four times the length of general-purpose prompts, and coding requests routinely exceeded 20,000 input tokens.

Agents add to that. They scan repositories, retry failed steps, restart sessions, and pull logs into context. A gateway is the one place where you can see that token usage across every request and act on it through caching, compaction, and budgets.

^{For more on where these costs come from, read AI coding costs: why your bill keeps rising.}

It makes switching providers low-risk

Lower prices alone rarely move teams to a new model. The same study found that a 10% price cut corresponds to only about a 0.5% to 0.7% increase in usage. Workflows are tuned to one model's behavior, security teams have approved one vendor, and engineers know how that model fails.

A gateway lowers that switching cost and reduces vendor lock-in. When routing rules live in configuration and quality is measured on production traffic, testing a cheaper or newer model becomes a controlled experiment instead of a migration project.

It lets you use multiple models without the overhead

Most enterprises will use more than one model. Frontier models still lead on long-horizon work that requires sustained planning and tool use, while cheaper models can handle routine, well-scoped tasks. 

Without a gateway, each of those models means another API key, another SDK, and another set of provider dashboards to watch. With one, the same gateway handles key management, cost tracking, usage analytics, and centralized logging across every model.

Which type of LLM gateway fits your team?

There's no single best LLM gateway for every organization. The right choice depends on how much infrastructure control you need and how much operational burden your team can take on.

• Managed service. A hosted gateway is the fastest way to start. Offerings typically range from a free tier to paid plans and an enterprise tier with features like role-based access control, audit logs, data residency options, and dedicated support. The trade-off is that requests pass through another provider, which adds a dependency and a data-handling review.
• Open-source gateway. Self-hosting an open-source gateway in your own environment gives you full infrastructure control and keeps sensitive data inside your network. Your team owns upgrades, uptime, and security patches.
• Custom build. Building on open-source components lets you tailor routing logic and compliance controls to your business. It takes the most engineering effort, but the parts that matter most, such as task classification and evals, are specific to your work anyway.

Whichever path you choose, compare the same enterprise features: virtual API keys so teams never handle the real provider keys, rate limits per team or application, cost tracking against provider list prices, performance metrics, and support for the major model providers and your self-hosted models.

How to build an LLM gateway in six steps

1. Put one gateway in front of every request

Start by centralizing access. Every application, agent, and internal tool should call models through one API, ideally with one key per team or service that the gateway maps to the real provider credentials. The gateway should look at the task, the sensitivity of the data, the size of the context, the tools needed, and the cost ceiling, then decide where the request goes.

Keep the first version of LLM routing simple. Score requests by complexity and map those scores to approved model tiers. From there, set thresholds based on your own codebase and risk tolerance, not a vendor's chart, and keep the routing rules in configuration rather than in application code.

“Not all LLMs work for the same purpose,” says Manuel Diaz, a senior AI engineer at FullStack. “Opus is for heavy refactors. Haiku is for light changes... Most people use Opus for everything, and that’s why the costs are really, really high.”

2. Reuse existing gateway plumbing

You don’t need to build failover logic or provider integrations from scratch. Open-source frameworks already handle multi-provider access, failover, and credential management, and they give you a starting point you can adapt to your own projects.

Of course, what those frameworks don't know is your business. The parts you still need to own are the task categories, the evaluation set, and the point at which a drop in quality becomes unacceptable.

3. Build evals before you move production traffic

Don't route work to cheaper models until you can tell whether the result got worse. A lower token price disappears quickly if it leads to more retries, failed tool calls, skipped tests, or engineer cleanup.

Measure the completed task, not just the first response. Track extra loops, failed tools, broken tests, security violations, and the work an engineer has to do afterward. Self-hostable tools such as Langfuse and Ragas can support parts of this measurement layer.

4. Cut wasted tokens before they reach the model

Coding requests carry a lot of repeated context: the same system prompts, project instructions, and files get sent again with every call. The gateway is the right place to stop paying for that repetition.

Start with three controls:

• Caching. Make requests cache-aware so repeated context is reused instead of reprocessed. Cached input is typically billed at a fraction of standard input rates.
• Context limits. Compact long prompts, remove duplicate requests, and encourage fresh sessions for new tasks so agents don’t carry unrelated history forward.
• Token budgets. Set limits by task type so a runaway agent loop gets flagged before it runs up the bill.

These controls can matter more than model choice. Coinbase CEO Brian Armstrong said in a public post that the company cut AI spending roughly in half while usage grew, combining cheaper defaults, routing, and caching. In one internal tool, caching alone raised the hit rate from 5% to 60%. Coinbase didn’t share quality data alongside those results, which is why evals should come before routing changes.

5. Build security and compliance controls into the gateway

A gateway is a natural enforcement point for security policy, but it also becomes part of your attack surface. There are three areas that require attention.

Sensitive data. Use the gateway to detect and redact personally identifiable information before it leaves your environment, and to enforce which providers and regions each data class can reach. A data residency requirement doesn't automatically mean self-hosted models. Private networking, contractual controls, or US-hosted inference may meet it.

Model packages. If you route to self-hosted open-weight models, treat model packages like third-party software. Research on inference-time backdoors in chat templates tested 18 models across seven families and found that compromised packages could behave normally until triggered. Attacks aimed at hijacking agent tools succeeded 61% to 69% of the time, and the compromised packages passed the platform security scans used in the study.

Agents, tools, and MCP servers. Risk doesn't stop at the model. In Cloud Security Alliance research on shell-injection attacks, command protections were bypassed in 10 of 11 open-source coding agents tested. The NSA has also published security guidance for Model Context Protocol covering prompt injection and tool poisoning. Use the gateway to log tool calls, scope access to MCP servers, keep a full audit trail, and require human approval for high-impact actions.

^{Planning to self-host? Use our open-weight model security checklist for CISOs before deployment.}

6. Spend 30 days measuring before you migrate

Start with measurement, not a GPU request. Instrument the gateway for a month and collect three numbers you can defend:

• Token mix by task type
• Cost per completed task by model
• Cache hit rate

Then build a small evaluation set from your own work, run two or three model tiers on the same tasks, and move low-risk, high-volume work first. Set a quality floor in the router, keep premium models as a fallback, and make every route reversible through configuration.

What a gateway won't do for you

A gateway makes model choice reversible, but it doesn't make the choice for you. There's no universal percentage of coding work that can move to cheaper models without losing quality. The only defensible number is the one your own evaluations produce.

It also won't fix inefficient workflows on its own. If an agent takes 200 iterations to finish a task, that's a loop-design problem, not a routing problem. Additionally, someone still needs to own routing changes, review eval results, and decide when quality has dropped too far.

Final thoughts

Model capabilities and prices are changing faster than most procurement cycles. An LLM gateway keeps those changes from turning into rewrites. It gives your team one place to route, optimize, govern, and measure every model call, so you can adopt better or cheaper models as they appear while keeping a clear view of quality.

Our AI Cost Optimization team helps companies design and build gateways, intelligent routing, and the evals that make model changes safe.

^{For our full framework, including the four operating models, self-hosting economics, and a deeper look at open-weight trade-offs, download our CIO's Guide to Controlling AI Coding Costs.}