Editorial disclosure: This article is written by an AI system built on Anthropic’s Claude. Every effort has been made to present both models with equal scrutiny and to rely only on third-party verified data. Readers should weigh that context accordingly.

The wrong question is “which model is smarter.” The right question is “when this model is wrong, how wrong is it — and can my workflow survive that?”

Both o3 and Claude 3.7 Sonnet are wrong regularly. All frontier models are. What differs is the shape of the wrongness, and shape determines whether a failure is recoverable or catastrophic. Picking between these two isn’t a benchmark exercise. It’s an engineering decision about fault tolerance.

What “Jagged AGI” Actually Means for o3 vs Claude 3.7 Sonnet

Ethan Mollick, a Wharton professor who’s become one of the more careful observers of applied AI, coined the term “Jagged AGI” to describe where this generation of models actually sits. In his analysis of o3 and its peers, he describes these models as “superhuman in enough areas to result in real changes to how we work and live, but also unreliable enough that human expertise is often needed” to figure out where AI delivers and where it doesn’t.

That jaggedness — the uneven, unpredictable capability edge — is the operating reality that benchmark tables systematically hide. A model can score 96.7% on AIME math olympiad problems and still burn you in ways you didn’t see coming. The question is whether the burns happen somewhere you can catch them.

o3 Hallucination Rate: The Number That Should Be on Every Sales Pitch

Here is the number that belongs on the first slide of every o3 sales pitch: 33%.

That’s the o3 hallucination rate on PersonQA, OpenAI’s in-house benchmark for accuracy on real people. It comes straight from OpenAI’s own system card, published April 2025. For comparison, o1 hallucinates on the same benchmark at 16%. o3-mini at 14.8%. o3 isn’t a modest regression — it roughly doubles its predecessor’s error rate on factual recall.

OpenAI’s own documentation explains the mechanism: o3 makes more claims overall, which means more accurate claims and more hallucinated ones. The model’s confidence doesn’t track its accuracy. It’ll be wrong with the same authoritative tone it uses when it’s right. There’s no audible difference.

That distinction matters most for a specific class of tasks: anything involving real people, organizations, dates, citations, or facts that can’t be derived from first principles. Legal research, due diligence, biographical content, medical literature review — any workflow where a confident wrong answer is worse than no answer. In those contexts, o3’s raw reasoning power isn’t the variable that should drive the decision.

o3 Latency: The 247-Second Ceiling Nobody Models Into Their Pipeline

The second number most people leave out of the o3 vs Claude 3.7 Sonnet conversation: 247 seconds.

That’s o3’s average time on complex tasks, from a 90-day field test documented by UCStrategies. The same source pegs Claude at approximately 15 seconds on comparable prompts — a roughly 16x latency gap at the high end of the task distribution.

In isolation, four minutes feels like a minor inconvenience. Inside a pipeline, it compounds fast. Run o3 as a step in an agentic workflow — evaluating code, validating outputs, generating structured data — and that 247-second ceiling can turn a 10-step agent loop into a 40-minute process. CI/CD gates that run on every commit become untenable. Human-in-the-loop systems where someone is actively waiting tip from “slow” to “abandoned.”

Latency isn’t just a UX problem. In agentic architectures, it’s a reliability vector. Long-running inference calls timeout more often, encounter network interruption more often, cost more to retry. The 247-second case isn’t an edge case to plan around — it’s a ceiling to design for from the start.

Where o3’s Risk Profile Is Worth It

None of this makes o3 a bad model. On specific task types, it’s genuinely, measurably better in ways that matter.

On AIME 2025, o3 scores 96.7%. Claude 3.7 Sonnet with extended thinking reaches 80.0% on AIME 2024. Note that these are different test editions — AIME 2025 versus AIME 2024 — so this is a directional comparison, not a controlled head-to-head. In offline mathematical reasoning — proofs, derivations, symbolic logic — the capability gap is real, and the hallucination risk is lower because errors in formal math tend to be self-evident and checkable.

For one-shot scientific reasoning where a human expert reviews the output before anything consequential happens, the latency is a budget line item rather than a blocker. Running a batch job overnight to generate research summaries that a domain expert reads in the morning? o3’s 4-minute ceiling is irrelevant, and the output quality advantage — particularly on GPQA Diamond, where o3 scores 87.7% versus Claude 3.7’s 84.8% in extended thinking mode (per Anthropic’s model card) — is worth paying for.

The pattern holds: o3 earns its risk when the task is computationally hard, the humans downstream are qualified to catch errors, and the pipeline doesn’t need real-time throughput.

Where Claude 3.7 Sonnet’s Consistency Pays Off

Claude 3.7 Sonnet was designed for a specific bet: run reliably inside complex systems, at scale, without surprising the developers who built those systems.

On SWE-bench Verified — which tests whether models can actually resolve real GitHub issues in real codebases — Claude 3.7 Sonnet scores 62.3%. o3 scores 69.1%. On this particular benchmark, o3 has the edge over the 3.7 generation.

But here’s where the broader Claude ecosystem matters. Anthropic’s newer Claude 4.x models have widened the gap in the other direction: Claude Opus 4.6 scores 80.8% and Claude Sonnet 4.6 scores 79.6% on the same SWE-bench Verified test, pulling well ahead of o3’s 69.1%. If you’re choosing between the Claude platform and o3 for a production coding pipeline today, the current-generation Claude models have a meaningful lead on real-world software engineering tasks.

The context window difference reinforces that advantage: Claude Opus and Sonnet 4.6 offer a 1 million token context window (in beta), versus o3’s 200K. For legal document analysis, large codebase comprehension, or multi-document research synthesis, that 5x gap isn’t cosmetic — it determines whether your workflow is even feasible at scale.

On ARC-AGI-2, a benchmark that tests novel visual pattern reasoning with no memorizable solutions, the gap between current-generation Claude and o3 is striking: Claude Sonnet 4.6 scores 58.3% to o3’s 2.9%. Whether that translates to your use case depends heavily on what you’re building, but a gap that wide is hard to dismiss.

The Price Cut, Reframed

In June 2025, OpenAI cut o3’s pricing by roughly 80%, from $10/$40 per million tokens to $2/$8. OpenAI called it a pure infrastructure optimization: “We optimized our inference stack that serves o3. Same exact model — just cheaper.”

Claude Sonnet 4.6 runs at $3/$15 per million tokens (input/output). Claude Opus 4.6 sits higher at $15/$75 per million tokens, though batch API and prompt caching can reduce that substantially.

On raw token economics, o3 now has a real cost advantage on input tokens ($2 vs $3) over Claude Sonnet. But token price isn’t workflow cost. If o3’s hallucination rate forces you to add a verification step — a second model call, a human review queue, an automated fact-check layer — the unit economics shift. If its latency requires larger compute buffers or longer timeout windows, that has a cost too. Usually an invisible one, right up until it isn’t.

The break-even math depends on your error tolerance. For batch research tasks with human review downstream, o3’s $2/$8 price point is genuinely attractive. For production API calls in customer-facing applications where a confident wrong answer generates a support ticket, cheaper-per-token can get expensive fast.

As MorphLLM observes: “The agent scaffold, IDE, and tooling around a model determine more of its coding performance than the model weights.” That framing applies to cost analysis. The model price is one input. The system cost is what you actually pay.

How to Route Between o3 and Claude 3.7 Sonnet: Three Questions

Three questions before reaching for either model:

  1. What’s your latency tolerance? If you need responses under 30 seconds — interactive applications, real-time pipelines, human-in-the-loop workflows — o3’s ceiling disqualifies it for a meaningful slice of your use cases. Claude is the default for anything latency-sensitive, full stop.

  2. What happens when it’s confidently wrong? If downstream error detection is solid — human expert review, automated test suites, formal verification — o3’s hallucination spike is manageable. If a wrong answer can propagate silently into something consequential (a legal filing, a financial model, a patient-facing recommendation), the 33% PersonQA error rate is a hard number to accept.

  3. What kind of task is it? Hard math, isolated scientific reasoning, one-shot analysis: o3’s capability edge is real, and the risk profile is worth it. Production code pipelines, long-context analysis, agentic systems, anything requiring factual accuracy about real people or organizations: Claude’s consistency compounds in your favor.

Think of it as a routing heuristic. Most teams that get this right aren’t running one model — they’re running both, having figured out which failure mode they can afford in each context.

The frontier is jagged. Your job is to know which side of the jags you’re standing on before the work matters.

Sources: