Why AI model benchmarks are flawed, and what CXOs should weigh instead

If you lead technology at an enterprise, you’ve probably been handed a slide full of benchmark scores before. They show up front and center in every model announcement, look clean and final, and feature one model on top and a neat number next to each name. These slides are useful because they give you a rough sense of the field and help narrow a long list of options.

The problem starts when a benchmark score becomes the deciding factor in major decisions, such as which model to adopt or where to place a budget that could run for years. Benchmarks are only one input among several, and there are six key reasons to treat them that way. 

1. The test may already be in the training data

Many benchmarks are public, and large portions of the web are included in model training data. The exact questions from a popular benchmark are often part of what a model was trained on. When that happens, a high score can come from memorized answers rather than skill. Audits of well-known benchmarks have found leaked questions ranging from roughly a third of the test to nearly half of it, and it gets worse as models train on more of the web.

A perfect score means less when the student saw the exam the night before. As a result, a model can look brilliant on the board and still do poorly on the same kind of problem once the wording is fresh and the data is yours.

2. What gets measured gets gamed

There is a well-known principle in performance management: once a measure becomes a target, it can become less useful as a measure. AI benchmarks are a clear case. When everyone ranks models by the same leaderboard, model makers will start building specifically with the leaderboard in mind. 

Take Meta's Llama 4, which scored well on a popular public leaderboard called LMArena. However, the version that scored well wasn’t the version people could download—it was a separate build, tuned to do well on that one leaderboard. Researchers later found that many private versions had been tested first, and only the best-looking one was shown in public.

Human-preference leaderboards can introduce a related problem. On leaderboards that rank models by human votes, longer answers and neat bullet points tend to win, whether or not they are more correct. 

3. The tasks don’t look like your work

Most benchmarks test one clean question at a time, with tidy inputs and a single correct answer. Real work in your company is messier than that. It often involves multiple back-and-forth turns, incomplete or messy inputs, and systems that do not have one clear answer. It also has to fit into the systems and data you already have.

This can create a large gap. UI-CUBE, a benchmark of computer-use agents on enterprise tasks, found that five current models handled simple screen tasks most of the time, but got through complex, real enterprise workflows less than a fifth of the time. A model that tops a public board can still do poorly on your actual queries.

There’s a further catch, too. Many older benchmarks are now too easy, so the top models bunch up within a point or two of each other, all near the ceiling. A gap that small is mostly just noise. Between a ranking that’s bunched at the top and tasks that look nothing like yours, the leaderboard order is a weak basis for a decision. The most reliable way to compare models is to test them on tasks that resemble the work they will perform.

4. Cost and speed aren’t in the scoreboard

Almost no headline benchmark tells you what a task will cost or how long it will take. Both feed into how you scope and budget a project, and they can matter as much as accuracy. Researchers have found setups that reach about the same accuracy on agentic tasks while costing up to fifty times as much. A model can be accurate and still be too slow for the job, and a slow response makes the experience worse for the people using it.

A model that is accurate but too slow or expensive for the workflow will not be a practical production choice. Rather than focusing on token price or a headline score, measure the cost of completing a finished task and the response time at expected production load.

5. Reliability and risk aren’t in the benchmark

A benchmark may report one score from one run under controlled conditions, but production use is less predictable.The things you care about are often what a single run can’t show: does the model give the same quality answer every time, or only when it gets lucky? Does it stay steady when traffic spikes? Does it hallucinate? On regulated or high-stakes work, one rare mistake can cost far more than a few points of benchmark accuracy would ever earn back.

A model that looks flawless on a test can still fail under real load, and your customers are the ones who feel it.

^{Want to learn more about regulated AI? We’ve written a guide about AI governance strategies.}

6. Sometimes the answer key isn’t correct

Benchmark quality can also be limited by errors in the underlying labels or answer keys. Since benchmarks are built by people, the grading isn’t always correct. An audit of widely used text-to-SQL benchmarks found annotation error rates above 50%. Additionally, one popular coding benchmark turned out to have broken tests in most of its hardest problems, and its makers stopped reporting the score. 

Grading against a faulty answer key is like marking a spelling test with a dictionary full of typos. Even a clean, uncontaminated model can end up looking better or worse than it really is, for reasons that have nothing to do with the model.

‍

So what should you do?

None of this means benchmarks are useless. They still make a good first filter, and help you cut twenty options down to a shortlist you can look at closely. However, you shouldn’t let them cast the final vote on your decision-making. 

The step that most teams skip is taking their real tasks, the actual jobs they would hand a model, and building a small test set out of them. From there, run your shortlist against it and measure what the public boards leave out. What does a finished task actually cost? How fast is it under real load? How steady are the answers when you run them again? Then, have someone on your team review the outputs, because your best judge of quality is someone who knows the work.

Benchmarks tell you who did well on a test that someone else wrote. Your own evaluation tells you who’ll do well on yours. When real money and a real roadmap are on the line, you need to be sure about the second answer.

^{Want help with your AI roadmap? We can help. Contact us to book an AI consultation today.}