Table of Contents
AI Success Rate Reliability Kids Can Understand With Dice
AI success rate reliability kids can grasp: a model solved a famous math problem in half its runs. A dice-and-coins lesson on why identical prompts differ.
Teaching AI success rate reliability kids can actually use starts with one number. In May 2026, an OpenAI model disproved a mathematical conjecture that had stood since 1946. OpenAI’s Sébastien Bubeck said the model produced correct solutions in about 50% of trial runs on the same prompt.
Same question. Same model. Half right, half wrong. And nothing in the output announces which one you got.
That property has a name in machine learning, and it is worth teaching with dice before you teach it with vocabulary.
Key Takeaways
- The model that disproved the Erdős unit distance conjecture produced correct solutions in roughly 50% of identical runs, per OpenAI.
- This is by design. Language models sample from probability distributions, so repeated runs differ even with the same input.
- A 50% per-run rate means that if you run it three times and take any correct answer, you succeed about 87.5% of the time. Sampling changes the math.
- Verification, not generation, is the bottleneck. A 50% generator is useful only to someone who can tell the two halves apart.
- The same property shows up in your kid’s homework chatbot. Asking the same question twice is a free experiment.
Why AI success rate reliability kids see is not a bug
A language model does not look up an answer. It produces text one token at a time, and at each step it has a probability distribution over what could come next. Unless you force it to always pick the single highest-probability token, it samples. Different samples mean different paths, and in a long chain of mathematical reasoning, different paths can lead to different conclusions.
That is the mechanism. No mystery, no magic.
For a short factual question, the paths mostly converge and you get the same answer every time. For a 100-page proof, they do not. Each step compounds, and a wrong turn at step 40 poisons everything after it. The longer and more open-ended the task, the more variance you see.
The Science News account of the Erdős result is careful about what we do and do not know here: the data supporting the 50% claim “have not been released or peer-reviewed” and the computational time was not revealed. Melanie Matchett Wood raised a related concern about OpenAI not disclosing failed attempts. So the 50% number itself is a company statement, not an independently verified measurement, which is a useful thing for a kid to notice too.
Trials, successes, and what sampling does to the odds
Here is the table I would draw for a 12-year-old. Assume each run succeeds independently with probability 0.5, and that you can tell a correct proof from an incorrect one when you see it.
| Number of runs | Chance at least one succeeds | Chance all fail | What it costs |
|---|---|---|---|
| 1 | 50% | 50% | 1 unit of compute |
| 2 | 75% | 25% | 2 units |
| 3 | 87.5% | 12.5% | 3 units |
| 5 | 96.9% | 3.1% | 5 units |
| 10 | 99.9% | 0.1% | 10 units |
The independence assumption is doing a lot of work here, and I want to flag it honestly. Real model runs are not fully independent; they share biases, so a model that misunderstands the problem may fail the same way ten times. Treat the table as the best case.
But the structure of the insight holds, and it is the important part: a 50% generator plus a reliable checker is a strong system. A 50% generator with no checker is a coin flip. That is why the verification question matters more than the capability question. Our piece on how mathematicians verify a 125-page AI proof walks through what checking actually involves.
How to Teach Your Kid About Stochastic Output
All three activities use dice, coins, or a chatbot you already have.
Ages 5–8: The Same Question Twice
Roll one die and write down the number. Roll it again and write that down. Ask your child why the same action gave two different answers.
Then do the equivalent with a chatbot if you have one: ask it the same question twice, in two separate conversations, and read both answers aloud. Point at the differences. “Same question, different answer” is a concept a six-year-old can hold, and it inoculates them against treating a chatbot like a fact machine.
Ages 9–12: Coin Flips and Best-of-Three
Give your child a coin. Heads means the AI got it right, tails means wrong. Flip once: they will see it is a coin flip. Now flip three times and count how often at least one head appears. Do it twenty times and tally.
They should land somewhere near 17 or 18 out of 20, which is close to 87.5%. Then the punchline question: what do you need in order to use best-of-three? The answer is a way to tell heads from tails, which in the AI case means someone who can check a proof. Without the checker, three tries gets you nothing.
Ages 13+: Run the Experiment on Homework
Have your teen pick a math problem they already know the answer to. Ask a chatbot the same question in five fresh conversations. Log each answer and whether it was right.
Most students expect 5/5. What they usually get is 4/5 or 3/5 on a hard problem, and sometimes five different approaches with different final answers. Then have them compute their own measured success rate. That number, generated by their own experiment, will stick better than anything you tell them about AI limits.
The question to ask: “If you can’t tell a right answer from a wrong one, how many tries does it take to be sure?”
Where else this shows up
The reliability question is not confined to math proofs.
Homework. Common Sense Media’s “Teens in the AI Era” survey, published August 18, 2026 with 1,017 US teens aged 13 to 17, found 70% use AI for schoolwork, with 25% using answers as-is, 31% rewriting them to sound like themselves, and 35% improving them with their own knowledge. Only 27% said a teacher ever discussed safe AI use with them. The 25% using output as-is are the ones exposed to per-run variance.
Benchmarks. At IMO 2026, multiple systems scored 42/42, but the harness matters. Two were graded by IMO organisers; four others were tested through a self-administered harness graded by AI agents. A perfect score reported with no attempt count is a much weaker claim than one reported under a fixed submission window.
School assessments. PISA 2025 found that 46% of OECD students use AI chatbots weekly to help them learn, and that students who use AI for specific tasks like summarizing score lower in science on average than non-users. Association, not causation, but the students most exposed are the ones who cannot evaluate what comes back.
What to actually do at home
Make “ask twice” the default
If a chatbot answer matters, have your child ask again in a fresh conversation. Two identical answers is weak evidence of correctness. Two different answers is strong evidence that one is wrong.
Teach the word “draft”
The most useful reframe I have found is calling AI output a draft rather than an answer. Drafts get checked. Answers get submitted. The vocabulary does real work.
Ask for the check, not the result
When your kid uses AI, the interesting question is not “what did it say” but “how did you confirm it.” If the answer is “it sounded right,” that is the gap.
Use the reliability number as a career argument
A tool that is right half the time makes the person who can tell the difference more valuable, not less. That is a more honest and more motivating framing than “AI will take your job.” We work through the labor data in whether AI solving math problems means math careers are over.
What not to do
Do not use the 50% figure to argue that AI is unreliable in general. It is a per-run figure on a frontier research problem, which is close to the hardest possible test. On routine tasks these systems are far more consistent. The lesson is about variance and verification, not about dismissing the tool.
What to Watch For Over the Next 3 Months
- Week 4: Does your child ask a chatbot the same question twice when the answer matters? That single habit covers most of the ground.
- Month 2 red flags: Submitted work your child cannot explain. That is the signature of accepting a single run without checking.
- Month 3 self-check: Ask your child what their own measured success rate was on the five-run experiment. If they remember the number, the concept landed.
Frequently Asked Questions
Why does AI give different answers to the same question?
Because it samples from a probability distribution at each step rather than looking up a stored answer. Different samples produce different reasoning paths, and on long tasks those paths can end in different conclusions. Short factual questions usually converge; long open-ended ones do not.
What does a 50% success rate actually mean?
That across repeated runs of the same prompt, roughly half produced a correct result. It says nothing about whether a specific output is correct. OpenAI reported this figure for the model that disproved the Erdős unit distance conjecture, though the underlying data have not been published or peer-reviewed.
Does running it more times fix the problem?
Partly, if you can tell right from wrong. At 50% per run and assuming independence, three runs give about an 87.5% chance that at least one is correct. But you still have to identify which one, so the checker is doing the real work.
Is my kid’s homework chatbot this unreliable?
On routine problems, no. The 50% figure applies to an unsolved research-level problem, which is the hardest case. But variance exists at every difficulty level, which is why asking the same question twice is worth doing when the answer matters.
How do I explain this without making my kid distrust everything?
Frame it as a property, not a flaw. A ruler is exact; a chatbot is probabilistic. Both are useful when you know which you are holding. The goal is a kid who checks, not a kid who refuses to use tools.
About the author
Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.
Sources
- Hulick, K. (2026, June 8). “AI guardrails and the Erdős math problem.” Science News. https://www.sciencenews.org/article/ai-guardrails-erdos-math-problem
- OpenAI. (2026, May 20). “An OpenAI model has disproved a central conjecture in discrete geometry.” https://openai.com/index/model-disproves-discrete-geometry-conjecture/
- Common Sense Media. (2026, August 18). “Teens in the AI Era: Schoolwork and Skills That Matter.” https://www.commonsensemedia.org/research/teens-in-the-ai-era-schoolwork-and-skills-that-matter
- OECD. (2026). “PISA 2025 Results (Volume I): Future-Ready Students,” Chapter 4. OECD Publishing. https://www.oecd.org/en/publications/pisa-2025-results-volume-i_73451bc5-en.html
- Digital Applied. (2026). “IMO 2026 perfect scores and AI benchmark saturation.” https://www.digitalapplied.com/blog/imo-2026-perfect-scores-ai-benchmark-saturation
- Tao, T. (2025, January). “Machine-Assisted Proof.” Notices of the American Mathematical Society, 72(1), 6–13. https://www.ams.org/notices/202501/rnoti-p6.pdf
- Kakaes, K. (2026, August 3). “Why the legendary Erdős problems are falling to AI.” Quanta Magazine. https://www.quantamagazine.org/why-the-legendary-erdos-problems-are-falling-to-ai-20260803/
