Benchmark Saturation: What a Maxed-Out AI Test Can't Tell You

Benchmark saturation is what happens when a test stops carrying information because too many things pass it perfectly. Not because the test was bad. Because it got solved.

At the 2026 International Mathematical Olympiad in Shanghai, July 10–21, multiple AI systems were officially graded at a perfect 42/42, including Huawei’s Celia and RedNote’s dots-note-3.0. Several other labs reported 42/42 under self-administered conditions. A year earlier, the best AI results were 35/42.

Here’s the thing a parent should notice: among 666 human contestants, exactly 7 scored 42. The test still discriminated brilliantly among humans and stopped discriminating at all among frontier AI systems. Same test, two very different information contents, in the same three weeks.

Key Takeaways

• A saturated benchmark is one where multiple systems hit the ceiling, so the score can no longer rank them. The information is gone even though the number looks impressive.
• At IMO 2026, Huawei’s Celia and RedNote’s dots-note-3.0 were officially graded 42/42, with several labs reporting 42/42 self-administered. Sources differ on the exact count, so “multiple” is the accurate word.
• Among humans, 55 golds were awarded at a 29-point threshold and 7 of 666 contestants scored 42. The same test remained highly informative for people.
• Saturation is a signal to change what you measure, not to celebrate. One analysis of IMO 2026 put it plainly: once several models share the ceiling, “the score itself stops carrying information.”
• Replacements are getting more specific: IMO-Bench with 400 answer-verifiable problems, 60 proof-based problems, 1,000 grading examples, and 60 Lean-formalized problems; plus task-level custom evaluations on real workloads.

Why a perfect score means less than a good one

Start with what a test is for. A benchmark exists to rank things: which model is better at this, and by how much. Ranking requires spread. If every candidate scores 100%, the test has told you that they all clear a bar and nothing about who’s ahead.

This is a familiar problem in education. If every student in a class gets an A, the grade stops predicting anything about who understood the material best. The teacher hasn’t learned less about the students in some cosmic sense; the grade has stopped being a useful summary.

One detailed analysis of the IMO 2026 results describes saturation as occurring when there are “multiple labs, multiple methods, multiple perfect scores” on a flagship evaluation, causing “the score itself [to stop] carrying information.” Its framing: “once several models share the ceiling on a flagship eval, the buying decision moves to four questions the headline number cannot answer.”

The accounting from that analysis, which is the most careful public tally I found:

Officially IMO-graded at 42/42: Huawei Celia and Xiaohongshu’s dots-note-3.0. Self-administered claims at 42/42, run under a harness and graded by a Claude agent: Claude Fable 5, GPT-5.6 Sol, Kimi K3, and Axiom Math’s AxiomProver. NVIDIA’s Nemotron 3 Ultra scored 30/42, IMO-team graded, above the gold threshold.

Note the two tiers. Official grading and self-administered grading are not the same claim, and the research log for this batch flags that sources vary on the count. South China Morning Post’s July 22 report focused on RedNote’s dots-note-3.0 as the first to reach a perfect score. Saying “multiple systems” is honest; naming an exact number is not.

What the humans’ numbers show by contrast

The official IMO 2026 results record 666 contestants from 117 countries, 55 gold medals at a 29-point threshold, and 7 perfect scores.

Do the division. About 1% of human contestants maxed the test. These are the strongest young mathematicians from 117 countries, working by hand, under time limits, with no verifier and no retries. The test discriminated finely among them: golds, silvers, bronzes, and a handful of perfect scores at the very top.

That contrast is the whole lesson, and it’s a better one than either “AI beat humans” or “the test was easy.” The same six problems produced a rich distribution among humans and a flat ceiling among frontier AI systems. Which means the test measures something real about human mathematical ability and something that has been substantially solved in AI systems.

What has been solved, specifically? Producing correct solutions to competition problems with verifiable answers, given large amounts of thinking time. The mechanism is documented: Snell et al. (2024) showed that compute-optimal allocation at answer time can beat naive sampling by more than 4x. Our explainer on test-time compute covers it, and what a 50% success rate means covers the reliability question that a single score hides.

What hasn’t been solved: posing worthwhile problems, recognizing which unsolved question matters, reliability across repeated attempts, and mathematics where the answer can’t be checked automatically. Quanta’s August 3, 2026 survey documents how much human verification still sits around every headline result, quoting mathematicians who reviewed the proofs themselves.

What replaces a saturated benchmark

Once a test saturates, the field builds harder or more specific tests. Three directions are visible right now.

More granular benchmarks in the same domain. The IMO 2026 analysis names IMO-Bench from Google DeepMind: 400 answer-verifiable problems, 60 proof-based problems, 1,000 grading examples, and 60 problems formalized in Lean. The design is telling. Splitting answer-verifiable from proof-based problems separates two skills that a single score conflated, and Lean formalization makes proof-checking mechanical.

Formal verification. Lean turns a proof into code that compiles or doesn’t, which converts a matter of expert judgment into a machine check. That matters because the alternative is human grading, and human grading is precisely what the 666 contestants at IMO 2026 received. Our piece on proof assistants and how math becomes code covers it.

Task-level custom evaluations on real workloads. The honest answer to “which model is best for my job” turns out to be: test it on your job. Anthropic’s September 1, 2026 Fable 5.1 release reported 52.6% on Terminal-Bench-Science 0.1 against 24.7% for Fable 5 and 29.0% for Opus 5. Notice that those numbers are nowhere near the ceiling. A benchmark in its useful range spreads results out like that, and a version number in the benchmark’s name (“0.1”) signals the designers expect to keep raising the bar.

There’s also a caveat worth stating: a score can be inflated by contamination, meaning the test problems appeared somewhere in training data. That’s a live concern with any widely published benchmark, and it’s one more reason a fresh, private evaluation on your own tasks beats a headline number.

How to Teach Your Kid About Benchmark Saturation

This is a measurement-literacy lesson, and it transfers to grades, sports stats, and product reviews.

Ages 5–8: The too-easy quiz

Give your kid a five-question quiz they’ll definitely ace (what color is the sky, how many legs does a dog have). They get 5 out of 5. Then ask: “If your friend also got 5 out of 5, who’s better at this quiz?” They’ll say you can’t tell. Exactly. Name it: “When a test is too easy, everybody gets the top score and it stops telling you anything. Then grown-ups have to make a harder test.” Then have them invent one question hard enough to separate them from a sibling.

Ages 9–12: Make the test harder on purpose

Have your kid design a ten-question quiz for the family on something they know well. Run it. Look at the scores. If everyone got 9 or 10, ask them to redesign it so the scores spread out. This is genuine test design, and doing it once makes the concept permanent. Then tell them the real numbers: 7 of 666 humans got a perfect score at the IMO while several AI systems all got perfect scores, and ask which group the test still works for.

Ages 13+: Audit a benchmark claim

Have your teen pick any AI model announcement and find a benchmark number in it. Then answer four questions: What is the maximum possible score? How close is this to it? What other systems’ scores are reported alongside? Is the benchmark version-numbered, and what does that suggest? Scores in the 20 to 60 range on a versioned benchmark are informative. Scores at or near the maximum are not. This is the skill that makes the rest of AI news readable.

The question to ask: “If two things both get a perfect score, how would you figure out which one is better?”

Benchmark status: what’s still informative

Benchmark	What it measures	Status as of late 2026	Why
IMO 2026 (official grading)	Competition math proofs, human conditions	Saturated for frontier AI	Multiple systems at the 42/42 ceiling; 7 of 666 humans
IMO 2026 (human contestants)	Same problems, by hand, timed	Highly informative	Full distribution: 55 golds at a 29-point threshold, 7 perfect
IMO-Bench (DeepMind)	400 answer-verifiable, 60 proof-based, 60 Lean-formalized problems	Designed post-saturation	Splits skills a single score conflated
Terminal-Bench-Science 0.1	Agentic science tasks in a terminal	In useful range	52.6% for Fable 5.1 vs. 24.7% and 29.0%; wide spread
Lean-verified proofs	Whether a proof formally compiles	Structurally reliable	Binary and machine-checkable; no grader judgment
Your own task set	Whether it works on your actual work	Always informative	Private, uncontaminated, matches what you need

The last row is the one that matters for a family choosing a tool. Nobody publishes a benchmark for “helps my 12-year-old understand fractions without doing the work for her,” and that’s the only evaluation that answers your question.

What to do at home

Check the distance from the ceiling

Whenever your kid sees an AI score, ask what the maximum is. A 96% on a test where the best systems score 94 to 98 is noise. A 52.6% where competitors score 24.7% and 29.0% is a real difference. Ceiling distance is the single fastest way to judge whether a number means anything.

Look for the version number

“Terminal-Bench-Science 0.1” tells you the designers expect to revise it. Benchmarks with version numbers are being actively maintained against saturation, which is a good sign about the people running them.

Build your own three-task test

Pick three things your family actually uses AI for. Write them down. If cost is part of your decision, the published per-token pricing is a far better comparison axis than any benchmark score, because it doesn’t saturate. When a new model appears, run those three. It takes fifteen minutes and it beats every published benchmark for your purposes, because it’s private, uncontaminated, and about your work.

Separate “official” from “self-reported”

At IMO 2026, some results were graded by the competition and some were self-administered. Both can be honest and they aren’t equivalent. Teaching a kid to ask “who graded it?” applies to AI benchmarks, science claims, and school assessments equally.

What not to do

Don’t read saturation as “the test was worthless” or “AI is now better than humans at math.” The test remained highly informative for 666 human contestants in the same month. And a maxed-out score on verifiable competition problems says nothing about the parts of mathematics that involve choosing what to work on. Our piece on what a perfect Olympiad score actually proves works through that distinction carefully.

What to Watch For Over the Next 3 Months

• Week 4: Your kid can explain why everyone getting a perfect score makes a test less useful, and can name the distance-from-ceiling check.
• Month 2 red flags: They repeat benchmark numbers without knowing the maximum. Or they’ve concluded benchmarks are all meaningless, which throws out the useful ones.
• Month 3 self-check: Hand them two benchmark claims, one near the ceiling and one mid-range, and ask which tells you more. Correct answer with a reason is a pass.

Frequently Asked Questions

What does benchmark saturation mean?

It means enough systems reach the maximum score that the score can no longer rank them. One analysis of IMO 2026 described it as multiple labs and methods producing multiple perfect scores, at which point “the score itself stops carrying information.” The response is to build a more discriminating test.

How many AI systems got a perfect score at IMO 2026?

Huawei’s Celia and RedNote’s dots-note-3.0 were officially graded at 42/42. Several others, including Claude Fable 5, GPT-5.6 Sol, Kimi K3, and AxiomProver, reported 42/42 under self-administered conditions. Sources differ on the exact count, so “multiple” is the accurate description.

Does this mean AI is better at math than humans?

It means AI systems now score at the ceiling on competition problems with verifiable answers, given substantial thinking time. Seven of 666 human contestants also scored 42, working by hand under time limits. Competition math is one slice of mathematics, and posing good problems remains a human activity.

What replaces a saturated benchmark?

More granular tests in the same domain (IMO-Bench splits answer-verifiable from proof-based problems and includes 60 Lean-formalized ones), formal verification where a proof compiles or doesn’t, and custom evaluations on the actual work you care about.

How do I judge an AI benchmark number as a parent?

Three questions, in order: What’s the maximum? How far from it is this score? What do competing systems score on the same test? A score far from the ceiling with wide spread among competitors is informative. A cluster near the top is not.

---

About the author

Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. International Mathematical Olympiad. (2026). “IMO 2026, Shanghai, People’s Republic of China.” https://www.imo-official.org/editions/2026/
2. Digital Applied. (2026, July 23). “IMO 2026 Perfect Scores and AI Benchmark Saturation.” https://www.digitalapplied.com/blog/imo-2026-perfect-scores-ai-benchmark-saturation
3. South China Morning Post. (2026, July 22). “World’s first AI model to earn perfect score in maths olympiad comes from China’s RedNote.” https://www.scmp.com/tech/article/3361482/worlds-first-ai-model-earn-perfect-score-maths-olympiad-comes-chinas-rednote
4. MarkTechPost. (2026, September 1). “Anthropic releases Claude Fable 5.1 and Claude Mythos 5.1: 52.6% on Terminal-Bench-Science and 75% cheaper cache reads.” https://www.marktechpost.com/2026/09/01/anthropic-releases-claude-fable-5-1-and-claude-mythos-5-1-52-6-on-terminal-bench-science-and-75-cheaper-cache-reads/
5. Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters.” arXiv:2408.03314. https://arxiv.org/abs/2408.03314
6. Quanta Magazine. (2026, August 3). “Why the Legendary Erdős Problems Are Falling to AI.” https://www.quantamagazine.org/why-the-legendary-erdos-problems-are-falling-to-ai-20260803/
7. TechXplore. (2026, July). “AI systems match humans’ score in prestigious math contest.” https://techxplore.com/news/2026-07-ai-humans-score-math-contest.html