Hosted open-weight APIs vs. self-hosted LLM: How to choose

Open-weight models have closed much of the gap with frontier models, and their token prices are often far lower. For teams watching API costs climb, that raises a natural next question: should we run these models on our own infrastructure?

For most organizations, the answer is not yet. Using an open-weight model and hosting it are two different decisions. A hosted LLM API gives you most of the price advantage right away. A self-hosted LLM gives you complete control over where model weights live and how data flows, but it only saves money when you can keep expensive hardware busy around the clock.

What are open-weight models, and how can you run them?

An open-weight large language model publishes its trained parameters, or model weights, under a license that lets you download and run them. That's not the same as open-source models. Under the Open Source AI Definition 1.0, open-source AI must also provide information about its training data and code. Most popular models in this category give you the working model, but not a full view of how it was made.

Licenses also vary widely across all these models, so check the terms for commercial use, modification, and fine-tuning. For example, Kimi K3 is released under a custom Kimi K3 License rather than a standard open-source license.

Once you've chosen an open-weight model, there are two ways to use it:

These two paths sit alongside frontier models and fine-tuned proprietary models. Most engineering organizations will use many models, matching each workload to the path that fits it.

^{For a closer look at licensing, read Open-weight models vs. open-source AI: What enterprise buyers need to know.}

What self-hosting looks like in practice

Self-hosting lets you run models anywhere from a laptop to a data center. The right setup depends on model size and how many users the system needs to serve.

• A local model on your own machine. Desktop tools such as LM Studio let developers download a smaller model and run local inference in a desktop app, with no external servers involved. Basic use requires no command line, though LM Studio's CLI and local server are available for developers who want to connect internal tools. This is a good way to experiment, but not a production platform for multiple users.
• Enterprise deployments on your own hardware. Production self-hosting usually means GPU servers in an on-premises data center or in your own cloud account, often packaged as Docker containers and served through an OpenAI-compatible API. Managed self-hosting platforms can handle some of the operational complexity while the model weights stay in your controlled environment.
• Edge devices. Small models can run directly on devices, which removes network round trips for latency-sensitive tasks.

Model size and VRAM set the limits

In every case, model size and VRAM are the defining constraints. The model weights have to fit in GPU memory, with room left for the context window and runtime overhead. That's manageable for smaller models but quickly becomes expensive for larger models. Self-hosted systems also tend to need more than the model itself, such as a vector database for retrieval over your knowledge base, and separate models for tasks like image generation.

Why hosted open-weight APIs are usually the better first step

If your goal is cost efficiency, a hosted open-weight API is usually the first place to look. You get the price advantage of open-weight AI models without an initial investment in hardware, GPU memory management, or an on-call rotation.

Pricing still varies widely between hosts. In August 2026, the same DeepSeek model was listed at prices ranging from about $0.53 input and $1.05 output per million tokens to about $1.91 and $3.83, depending on the provider. That's a more than threefold spread for identical weights, so shop the host as carefully as you choose the model.

Where the model is served matters too. Many leading open-weight coding models come from China, and some teams are comfortable using the weights but not sending prompts, code, or telemetry to a provider outside the United States. US-based inference platforms offer many of these models, sometimes at a higher price, which can resolve that concern without self-hosting.

Why a self-hosted LLM usually costs more than it looks

Self-hosting looks like a way to swap a variable API bill for fixed infrastructure. For interactive workloads, the hidden costs usually make it more expensive, not less.

The hardware is large and expensive

The strongest open-weight models are also the hardest to run. Kimi K3, one of the top open-weight coding models, has 2.8 trillion parameters, and its weights take up roughly 1.4 to 1.6 TB, depending on how they're measured. No single H100, H200, or B200 GPU can hold it; it needs a multi-GPU node at minimum.

Utilization decides whether self-hosting pays

Real traffic rises during the workday, drops at night, spikes around releases, and sits idle. That makes utilization the biggest variable. As a directional example, a large model on a multi-GPU node might cost:

These figures are a sensitivity analysis, not a procurement quote. Actual costs depend on reserved capacity, throughput, batching, power, networking, engineering support, and negotiated terms.

A hardware commitment also competes with falling prices. According to Mozilla's State of Open Source AI report, the price of GPT-4-class inference fell roughly 60x over 45 months. Any GPU purchase has to beat where hosted prices are likely to go over the life of that hardware, not just today's rates.

The model that fits your hardware may be less capable

On the Terminal-Bench 2.1 leaderboard in August 2026, the best open-weight model available through an API trailed the leader by about five points. The best open-weight model that fits on a single 8-GPU node trailed by about 18 points. The models closest to the frontier are the ones most teams can't realistically run on their own hardware, so expect some output quality trade-off on complex tasks. Choosing the right model means balancing task performance against available GPU memory, then benchmarking upward only if your hardware can support it.

When self-hosting makes sense

Self-hosting rarely delivers significant cost savings on its own, but several situations tip the decision in self-hosting's favor:

• Strict data protection requirements. In regulated industries or air-gapped environments, sensitive data sometimes can't leave your infrastructure at all.
• Sustained, predictable volume. Steady, non-interactive workloads such as large batch jobs can keep hardware busy enough to approach break-even.
• Avoiding vendor lock-in. Once you've downloaded and retained a model, no provider can retire it, change its rate limits, or reprice it on you.
• Models built on proprietary data. For some companies, they may see a stronger long-term case for self-hosting when a model is fine-tuned on, or built around, their own business logic and knowledge base, and when they want to run multiple models and choose different ones for different workloads. That kind of machine learning work, from model training to fine-tuning, often benefits from keeping both data and weights in-house.

Data privacy doesn't always require self-hosting

If the requirement is to keep sensitive data within a defined network or jurisdiction, private networking, contractual controls, redaction, or US-hosted inference may meet it.

"I would only really want to do self-hosting if data privacy and sovereignty is the number one thing," says Brad DeFrank, Director of Technical Account Management for AI Enablement at FullStack. "We can find a way to do it where the data sovereignty and privacy is not compromised even if you don't self-host it."

Risks that apply to both paths

Model origin and policy. Many leading open-weight coding models come from China, and US policy is still evolving. In January 2025, the Commerce Department added Chinese AI entities to its Entity List, including Zhipu AI, and the FY2026 National Defense Authorization Act restricts the use of DeepSeek models in Department of Defense systems. Have legal and security teams review the weights, license, origin, and serving jurisdiction before building a critical workflow around a model.

Supply-chain security. Model packages can include code that controls how prompts are prepared and served. Research on inference-time backdoors in chat templates found that compromised packages could behave normally until triggered, and attacks aimed at hijacking agent tools succeeded 61% to 69% of the time. Lock approved versions, verify them, and restrict what the serving environment can reach.

Ecosystem maturity. Open-weight models often have fewer integrations, deployment paths, and documentation than frontier APIs. "The open-weight models do have a high degree of performance on coding tasks," DeFrank says. "The challenge is the ecosystem around them is not built." In Mozilla's survey of 1,410 developers, only 17% named model performance as a barrier to using open models, compared with 27% who cited infrastructure and compute costs and 26% who cited security, privacy, or compliance concerns.

^{Before you deploy, use our open-weight model security checklist for CISOs.}

How to choose the right path

1. Name the real requirement. Is the goal cost control, data control, independence from a provider, or a fixed workload? The decision-making process should weigh cost, data boundaries, flexibility, and the team's infrastructure capacity before picking a deployment path.
2. Start with a hosted API behind a gateway. A gateway makes model switching a configuration change. If a self-hosted node fails later, the system falls back to a hosted API instead of going down.
3. Measure for 30 days. Track token volume by task type, cost per completed task, and how steady your traffic really is.
4. Evaluate quality on your own work. There's no universal share of work that moves to open-weight models with minimal quality loss, so test hosted and candidate self-hosted models on the same tasks, and validate the right model against your workload and hardware limits.
5. Self-host only when the case is clear. If you have a hard boundary that hosted inference can't meet, or volume that keeps hardware busy, self-hosting becomes a serious engineering option.

Final thoughts

Open-weight models are a real lever for lowering AI costs, but the savings usually come from using them, not hosting them. Start with hosted APIs, own the routing and evaluation layer, and treat self-hosting as an infrastructure decision you make when your data boundary or volume justifies it.

Our AI Cost Optimization team helps companies compare hosted and self-hosted options, model the real costs, and build the systems that make switching safe.

^{For more information on open-weight models and their implementation, download the CIO's Guide to Controlling AI Coding Costs.}