OpenAI's Terra, Luna, and Sol vs. Anthropic: Which wins what?

The first time we ran this benchmark, nearly every score sat at or near 100% across three trials per model. It pointed to a tidy conclusion: the frontier is solved, the models are interchangeable, and the best vendor is a matter of preference. However, running it properly at ten trials per model per task—240 runs in all—produced a more accurate picture. 

The test pitted Anthropic’s Opus 4.8, Sonnet 5, and Haiku 4.5 against OpenAI’s new GPT-5.6 family, Sol, Terra, and Luna, and tasked each model with real enterprise work: merging access-control rules, untangling a noisy incident feed, writing an authorization engine with a full audit trail, and planning a zero-downtime database migration. Our grading was simple. Either the code passed a test, or it didn’t—and if a model ran past its output budget (16K tokens) mid-answer, that also counted as a miss.

A quick refresher on the four tests

The suite consists of four tests. Each one is real enterprise work where a subtly wrong answer means an outage, a security hole, or downtime, and each carries a weight, so the hard tasks count for more.

OpenAI’s three models passed all their tests without a single miss. Anthropic’s models passed 82% of the tests, but where they succeeded, they did at a fraction of the cost and speed of the OpenAI counterparts. The results make it clear that not everything requires the top model.

Weighted pass score per model (harder tasks carry higher weight)

Models only stumbled on the toughest (migration planner) task

Every model aced the calibration task and the authorization engine, and only Haiku stumbled on incident correlation. 

Most of the ranking, then, comes down to a single task: the migration planner, which puts the “multi-agent intelligence” of these frontier models to the test. Haiku couldn’t complete it, and Sonnet ran into the token cap seven out of ten times. Even Opus, despite its capabilities, failed twice.

However, the most surprising result was that Luna, OpenAI’s cheapest model, aced even the hardest task as consistently as its more expensive sister models. 

The final results are nuanced, but make the trade‑offs for enterprise teams clear

If there’s one result in this study that warrants attention, it’s Luna—OpenAI’s lowest-tier model. Not only did it complete all 40 tasks, but it also did it at a lower overall cost. While there will undoubtedly be work beyond Luna’s capabilities—namely, tasks requiring deeper reasoning or longer context performance—it performed comparably to both its sister models and its Anthropic counterparts across the test suite. 

It’s also important to note that our test is a single-shot benchmark. External evaluations focused on longer, multi-step agentic workflows typically place Sol and Terra ahead of Luna on extended, complex work, which is consistent with how OpenAI positions its tiers. 

From a top-tier model perspective, Sol beat Opus by around 7%. However, on closer inspection, Sol bought those seven points of reliability at roughly three times the cost per result and about nine times the latency. Opus answered in 23 seconds on average; Sol took over three minutes. For many flagship workflows, a correct result in 23 seconds at one-third of the price—with a rare migration slip—is a better tradeoff than a perfect result in three minutes. Of course, this won’t work for certain workloads, with this distinction being the real decision facing enterprise buyers.

For teams optimizing for cost and reliability, there’s one clear winner in our test suite: Luna by OpenAI.

Averaged across each vendor’s three models, the two companies divide the results across two different measures:

OpenAI’s lineup is more uniformly reliable across the suite. Anthropic’s is cheaper and faster where its models succeed. Ultimately, which of these matters more depends on the workload and the cost of a failed run.

Head to head, tier by tier:

FullStack’s honest recommendations

No vendor sweeps our evaluation on every axis. Our practical answer is to treat model selection as a routing decision.

1. For the lowest miss rate, reach for GPT-5.6—but pick Terra or Luna, not Sol. While all three OpenAI models hit 100%, Sol charges two to four times as much and runs far more slowly for the same result. Paying flagship rates for the kinds of tasks behind this report buys you latency, not additional reliability. 
2. For the best overall value, opt for Luna. At four cents per success, OpenAI’s fastest model flawlessly completed hard frontier tasks. If your workload consists of a high volume of repeatable calls, it’s the most efficient option available as of this post.
3. For the lowest cost per result where occasional failures are acceptable, Opus 4.8 is worth serious consideration. It was the fastest model tested—at one-third Sol's cost—and finished at 93% with occasional migration misses. For interactive, human-in-the-loop workflows where speed is a priority and a retry is cheap, that trade-off is likely worth it.
4. For migration-style planning work, handle Sonnet 5 and Haiku with care. Sonnet’s problem is specific and addressable: it consistently runs out of output budget on long plans rather than producing incorrect ones. A higher output ceiling will likely improve its score, though teams should account for its higher token cost.

Haiku’s failures run deeper. It generated code that didn’t execute, and a larger budget won’t resolve a syntax error. A compile-and-retry gate should be in place before you use it in any high-stakes pipeline.

Should you trust a result where no Anthropic model hit 100%?

That’s a fair question, and one we asked ourselves before publishing. Ultimately, there are several factors about the test that gave us confidence. 

The gaps are large rather than marginal: a model going three of ten, or zero of ten, on a task isn’t noise. The failures also carry consistent, identifiable signatures—truncation for Sonnet and non-running code for Haiku—which is characteristic of structural weaknesses rather than random instability.

OpenAI’s strength in agentic coding is also reflected in independent public benchmarks, so a clean sweep here isn’t an artifact of our test harness.

WHAT THESE QUICK BENCHMARKS ARE, AND WHAT THEY AREN’T

Ten trials across four tasks give us enough data to see consistent patterns on this suite, but four tasks is still a narrow, deliberately edge-heavy set—not a general capability ranking. Reliability is measured under a fixed 16,000-token output budget,  which is the single variable most likely to improve Sonnet’s mid-tier result if raised. Haiku ran under a slightly different reasoning configuration than the other models, with a fixed thinking budget rather than high effort, though that gap doesn’t explain its non-running code failures. This evaluation measures single-shot code generation, not full agentic sessions, which cost more and exercise different skills. Sonnet 5 is on introductory pricing through August 31, 2026. Model prices were confirmed against Anthropic’s and OpenAI’s live pricing pages on July 15, 2026.

FullStack runs these evaluations independently—across vendors rather than relying on any lab’s published launch benchmarks—because our team ships real software for real customers and integrates these models directly into our development process. Our takeaway isn’t that OpenAI wins or that Anthropic wins. Rather, on this suite, we found that OpenAI is the more reliable default and that Anthropic is the faster and more cost-efficient option.

The best approach is to route each job to whichever of those qualities matters most for that workload. Done well, your team will capture most of OpenAI’s reliability at most of Anthropic’s price, giving you the best of both worlds.