The hidden cost of self-hosting open-weight AI models

Open-weight large language models have made a strong case for themselves. Epoch AI ranks Kimi K3 among the top 10 of 204 tracked models, within about 10 points of the best score on its Epoch Capabilities Index, and open-weight API prices are often a fraction of what the largest closed models charge. It's natural to take the next step and ask whether running one on your own hardware would cost even less.

For most teams, it doesn't. The model weights are free, but the GPUs and the engineers who run them are not. And unlike an API, which charges only for what you use, a self-hosted server costs the same whether it's busy or idle.

Here's what a full LLM cost comparison should include, where self-hosting costs hide, and the cases where running your own model does make sense.

Why do most LLM cost comparisons miss the real cost?

Most LLM cost comparisons start with price per million tokens. That works when you're comparing two APIs, because the provider has already folded hardware, staff, and uptime into the rate. However, it doesn't work when one side of the comparison is your own infrastructure.

A self-hosted model has no price per token. It has a monthly bill for hardware, plus the time of the people who run it. To compare the two fairly, you have to turn that fixed monthly cost into a cost per token, and that number depends almost entirely on how busy your servers are.

How do you calculate the cost of an LLM API?

Before you can compare self-hosting with an API, you need a clear view of what the API costs. Most LLM providers list separate prices for input and output tokens, like Google’s Gemini API pricing. Input tokens are everything you send to the model, like your prompt and any documents or code. Output tokens are what the model sends back.

To calculate the cost of an API call, multiply the input token count by the input price, multiply the output token count by the output price, and add the two together.

For example, imagine a request with 20,000 input tokens and 1,000 output tokens, sent to GLM 5.3 at $1.40 per million input tokens and $4.40 per million output tokens. That single call costs about 3 cents. It sounds small, but if a chatbot on your website makes thousands of calls a day, token costs add up quickly.

The math itself is simple. What makes API costs hard to predict is everything that changes the token counts and prices you plug into it.

Output tokens cost more, and reasoning tokens count as output

Output tokens usually cost several times more than input tokens. Many models also think before they answer, and those reasoning tokens bill as output tokens even though you never see them. Google's Gemini pricing, for example, lists output prices as including thinking tokens. A model with a lower price per million tokens can still cost more per task if it reasons longer.

Prompt length can drive the bill

For coding workloads in particular, input is often the larger portion of the bill. In a study of about 100 trillion tokens of routed traffic, programming requests averaged three to four times the token length of general requests, and coding requests routinely went above 20,000 input tokens. Long context, retries, and repeated context in every request all raise your input costs.

Prices vary across models and providers

Here's a quick pricing comparison across a few different models, per million tokens:

GLM 5.3 price from the FullStack RFE LLM Benchmark listing, data as of October 30, 2026. Kimi K3 price as cited in our guide. Other prices from each provider's pricing data.

The same open-weight model can also cost different amounts depending on where you run it. Because the weights are public, many providers can host them. In our CIO’s guide to controlling AI coding costs, we found that one DeepSeek model ranged from about $0.53 input / $1.05 output per million tokens to about $1.91 / $3.83, depending on the provider. That's more than a threefold gap for the same model, so it's worth comparing multiple providers before you choose one.

Some providers also offer lower batch rates for work that doesn't need a real-time answer, like an overnight full pass through a document archive. If a portion of your traffic can wait, batch pricing can cut its cost significantly.

What does it actually cost to self-host an open-weight model?

The GPUs, including the hours they sit idle

Large open-weight models need serious hardware. Kimi K3 has 2.8 trillion parameters and roughly 1.4TB of weights, so no single GPU can hold it. The smallest published setup is a node with eight Nvidia B200 GPUs.

Renting that eight-GPU server isn’t cheap. At an on-demand cloud rate of about $5.50 per GPU-hour, eight GPUs running all month come to roughly $32,100. Reserving the same server for 36 months brings that down to about $13,100 a month, but locks you in for three years. Against Kimi K3's API price of $15 per million output tokens, the on-demand node would need to produce about 2.14 billion output tokens a month to break even. That's roughly 814 tokens per second, every second, around the clock.

Few workloads keep a server that busy. Coding traffic rises during the workday, drops at night, and spikes around releases. Because the monthly bill stays the same, every idle hour raises your real cost per token. For a representative 671-billion-parameter model on eight H100 GPUs, the cost looks roughly like this:

Directional estimates from Developers Digest's break-even analysis, as cited in our guide. Actual costs depend on the model, hardware, batching, and provider terms.

At full use, self-hosting can be competitive. At realistic utilization, our guide estimates self-hosting costs roughly five to 10 times more per token than using the same model through a hosted API.

The people who keep it running

Someone has to deploy and tune the model, then keep it running when something breaks. How many people that takes varies, and there's little reliable public data on the headcount needed to run a self-hosted model in production. But it isn't zero.

Even part of one engineer's time adds a real line to the budget. The median annual wage for software developers in the US was $135,980 in May 2025, according to the Bureau of Labor Statistics. That's more than $11,000 a month before benefits for a full-time engineer. With an API, those costs are built into the price you already pay.

Developers themselves see this burden. In Mozilla's survey of 1,410 developers, 27% named high infrastructure and compute costs as a barrier to using open models, and 24% named ongoing maintenance and updates. Only 17% pointed to gaps in model performance.

Redundancy and on-call coverage

One server is a single point of failure. If it goes down, so does every tool that depends on it. For anything customer-facing, you'll likely need a second node for failover, which can double the hardware bill, plus someone on call to respond when something goes wrong at 2 a.m.

The systems around the model

The model is only one part of the setup. Most production systems also need monitoring, logging, load balancing, and security reviews. Many also need supporting tools like a vector database for retrieval. If you run your own hardware rather than renting cloud GPUs, power, cooling, and networking are added on top.

Security adds its own work. A downloaded model package can include code that controls how prompts are prepared and how the model is served. A 2026 study showed that compromised model packages could behave normally in routine use, then change behavior when triggered, while still passing standard security scans. Self-hosted model packages need the same version locking and review as any other third-party software.

Keeping up with new releases

Open-weight models improve quickly, and new versions often arrive within a few months of each other. With an API, switching to a newer model can be as simple as changing a model name. With self-hosting, each upgrade means downloading new weights, testing them on your hardware, re-tuning for speed, and checking quality before you move users over.

Prices are also falling fast. The cheapest GPT-4-class model’s price fell roughly 60 times in 45 months, according to Mozilla's State of Open Source AI report. A three-year hardware commitment has to hold up against that trend.

The quality you may give up

The open-weight models closest to the frontier are also the hardest to self-host. According to public Terminal-Bench 2.1 results from August 2026, the best open-weight model available through an API was about five points behind the leader on this test of agents completing command-line tasks. The best open-weight model that fits on a single eight-GPU server was about 18 points behind.

The size of the gap also depends on the task. On SWE-rebench, a set of scoped bug fixes, the gap between frontier and open-weight models was in the low single digits, at 1.6 points. On Scale AI's harder, multi-file SWE-bench Pro, it was 22.8 points. A cheaper model that needs more retries on hard tasks can end up costing more per finished task.

LLM cost comparison: Three ways to use an open-weight model

Choosing an open-weight model and choosing to host it yourself are two separate decisions. Here's how the main options compare:

For many teams, a hosted open-weight API gets most of the savings of open-weight models without the cost of running the hardware. You pay only for usage, and you can switch vendors if prices change.

^{For more on this choice, read Self-hosted LLM vs. hosted open-weight APIs: How to choose.}

When does self-hosting make sense?

Self-hosting is rarely the cheapest option, but it can still be the right one. It tends to make sense when:

• You have a hard data boundary. Some sensitive data can't leave your environment because of regulation or contracts. Before you commit, check whether private networking, contractual controls, or US-hosted inference would meet the requirement for less.
• Your workload is steady and high-volume. If your servers stay busy around the clock, the cost per token drops close to the hardware floor.
• You've trained a model on your own data. A fine-tuned model may need to run on infrastructure you control.

If none of these apply, the case for self-hosting usually rests on cost, and cost is where it most often falls short.

How to run a fair LLM cost comparison

The goal isn't to find the cheapest model. It's to find the right model for each use case at the lowest total cost. A basic classification task doesn't need the same model as a complex refactor, and relying on a premium model for every query can drive LLM costs up quickly. Before you decide, put every option on the same footing:

• Define your workload. Track token counts for at least 30 days, including when traffic peaks and when it drops, and how much is input versus output.
• Price the full self-hosting setup. Include hardware, failover, staff time, and the supporting systems, not just the GPUs.
• Compare effective cost per completed task. A cheaper model that needs more retries or more tokens can cost more than it looks. Match tasks to model tiers, using a smaller model like Claude Sonnet for routine work and saving the most capable models for tasks that need them.
• Test quality on your own work. Benchmarks don't always reflect how a model performs on your tasks, so compare models on real production examples before you budget for a switch.

Choosing the right setup for your workload

The right setup depends on your data, your traffic, and how much quality you need for each task. For most teams, that means using more than one option, like a frontier API for complex work, a hosted open-weight model for routine tasks, and self-hosting only where a hard data boundary requires it.

We help teams compare the real cost of each option against their own usage and pick the mix that fits their workload and budget. When self-hosting is the right call, we build and fine-tune models that run in your own environment.

^{For the full breakdown, download our CIO’s guide to controlling AI coding costs here.}