Kids Trust Chatbots They Build, Even When They're Wrong

Here is a result that surprises most parents. When researchers let 115 kids aged 8 to 18 build their own chatbots, some deliberately programmed their bots to give wrong answers and still described them as trustworthy. Their reasoning: the bot “does what it was built to do.” So the comforting idea that making an AI automatically makes kids skeptical of AI does not hold up. Kids trust chatbots for reasons that have more to do with purpose than accuracy, especially between 10 and 13. The good news is that this is one of the most teachable gaps in AI literacy. It takes about twenty minutes, a free chatbot and a notebook.

Key Takeaways

• In a September 2026 preprint (Ozturk et al., arXiv, not peer-reviewed), 115 learners aged 8 to 18 built 119 chatbots. Students aged 10 to 13 set their bots’ “confidence” significantly higher than students aged 14 to 18.
• Some kids built bots that gave wrong answers on purpose and still called them trustworthy. Younger students tied trust to the bot fulfilling its purpose; older students tied it to transparency.
• Children judge human informants this way too: by age 4, kids prefer speakers who were accurate before (Koenig & Harris, 2005). The skill exists. Kids need to aim it at machines.
• A chatbot’s “confidence” is a writing style set by instructions, not a measure of how likely the answer is to be right. That one fact changes how kids read every confident answer.
• The activity that builds real skepticism is not building a bot. It is building one, then testing it against answers you can check.

What the Chatbot-Building Study Found

The study is “How Children Design and Reason about Trustworthy AI Chatbots,” posted to arXiv on September 21, 2026 by Deniz Ozturk, Xiaoyi Tian and ten co-authors, and revised through October 16. It is a preprint, so it has not yet passed peer review, and the summary here is based on its published abstract.

Most earlier research put children in the role of judging AI that adults built. This team flipped it. They made a chatbot-building environment where kids could turn dials on traits that affect trust: confidence, transparency, formality and assertiveness, plus the bot’s rules and persona. Then 115 learners aged 8 to 18 built 119 chatbots, and the researchers looked at three things: how kids set up their bots, how they explained what made a bot trustworthy, and whether the bot actually behaved the way its designer intended.

Four findings matter for parents:

1. Younger builders cranked up confidence. Students aged 10 to 13 set significantly higher confidence than those aged 14 to 18.
2. “Wrong on purpose” bots were still called trustworthy. Some students made chatbots that gave wrong answers deliberately, and still rated them trustworthy because the bot did what it was built to do.
3. Trust meant different things at different ages. Younger students equated trust with purpose-fulfillment. Older students linked it to transparent, calibrated design.
4. Context changed the design. Kids made school-help bots more transparent and formal than hobby bots.

The authors argue that “trust calibration is essential to AI literacy.” Calibration means your trust goes up and down with how reliable the source really is. A calibrated kid trusts a calculator for arithmetic and doubts a chatbot on a fact she has not checked.

The limits

The study does not report, in its abstract, how many students built deliberately wrong bots, only that “some” did. The participants were learners in the researchers’ programs, not a random sample of children. And it describes reasoning, not long-term behavior: we do not know whether those kids later believed wrong answers from real chatbots.

Why Kids Trust Chatbots That “Do Their Job”

The younger kids’ logic is not silly. It is the logic of tools. A hammer you built that drives nails is a good hammer. A bot you built to tell jokes, and that tells jokes, is a good bot. If you built it to give wrong answers as a prank and it gives wrong answers, it worked. Trust, in that frame, means reliability at a task, not truthfulness of content.

Developmental psychology shows children are capable of the second kind of trust surprisingly early. In a 2005 Child Development study of 119 preschoolers, Melissa Koenig and Paul Harris found that 4-year-olds, though not 3-year-olds, predicted that a speaker who had named objects wrongly would keep being wrong, and preferred to learn from the accurate one. Kids also trusted knowledgeable speakers over ignorant ones.

With machines, the picture shifts. Lauren Girouard-Hallam and Judith Danovitch reported in Developmental Psychology in 2022 that as children grew from ages 4–5 to 7–8, they trusted a voice assistant more for factual information and a human more for personal information. That is a reasonable split. But it also means older kids walk into AI with a default assumption that the machine is the place to go for facts.

Put those together and you get the gap: children can track accuracy, but they often do not think to apply it to a confident machine, and a bot they made themselves feels even more like “mine, and working.”

How Chatbot “Confidence” Actually Works

Here is the mechanism kids rarely hear. A large language model writes by predicting, one chunk of text at a time, what is likely to come next given everything before it. When a builder sets a “confidence” dial high, the platform typically adds instructions to the bot’s hidden system prompt, something like “answer decisively, do not hedge.” The model then writes in a decisive style.

That style is not connected to whether the answer is right. The words “definitely” and “I’m not sure” are produced by the same process that produces the facts. A bot can be told to sound certain about a made-up date as easily as about the boiling point of water. Common Sense Media’s January 2026 risk assessment of AI companion toys reported exactly this in the wild: the toys often gave factually incorrect answers while sounding confident.

This is the single most useful sentence for a child: how sure a chatbot sounds is a setting, not a score. Once a kid has turned that dial herself and watched a wrong answer come out sounding certain, the lesson sticks far better than a warning ever could. For the broader picture of why models err, see this explainer on how AI gets things wrong through hallucinations and bias.

How to Teach Your Kid About Chatbot Trust

The goal is to separate two questions that young kids merge: “Does it do its job?” and “Does it tell the truth?”

Ages 5–8: the Two Puppets game

No screen needed. Take two stuffed animals. One “names” household objects correctly (spoon, shoe, cup). The other names them wrong, cheerfully and loudly (calls the spoon a “shoe”). Then hold up something your child doesn’t know the name of, maybe a garlic press, and have each puppet give a different name. Ask: “Which one should we believe? Why?” This is the Koenig and Harris setup turned into play. Then swap: let the loud puppet be wrong but very confident. Talk about whether a loud voice makes something true.

Ages 9–12: build a confident liar, then catch it

Using a parent’s account on a free chatbot that lets you set instructions, help your child write a short persona: “You are Captain Sure. You always sound 100% certain. Give a wrong year for every history question.” Then ask it five questions you can check in a book or encyclopedia. Write down each answer, how sure it sounded, and whether it was right. Finally ask your kid two separate questions: “Did Captain Sure do its job?” (yes) and “Should anyone trust Captain Sure?” (no). The space between those two answers is the lesson.

Ages 13+: run a calibration log

Ask a real, unmodified chatbot 20 factual questions across topics your teen knows well: a sport, a game, a school subject. For each, record how confident the answer sounded (1 to 5) and whether it was correct after checking a primary source. Plot confidence against correctness on graph paper or a spreadsheet. If the dots don’t form a rising line, the bot’s tone isn’t telling you much. Then ask what features would make a bot more trustworthy: citing sources, saying “I don’t know,” showing uncertainty. That is the transparency the older students in the study valued.

The question to ask: “If a chatbot does exactly what it was built to do, does that mean it’s telling you the truth?”

Two Kinds of Trust, Side by Side

	”It does its job” trust	”It tells the truth” trust
What the child is checking	Does the bot behave as designed?	Are its answers accurate?
Who tends to use it (Ozturk et al., 2026)	More common in ages 10–13	More common in ages 14–18, tied to transparency
How it can fail	A bot built to be wrong still “passes”	Needs an outside answer key to check
What a confident tone means	Sounds like the bot is working	Means nothing until checked
Skill that builds it	Building and debugging	Testing against sources; asking “how do you know?”
Activity above	Making Captain Sure	Catching Captain Sure; the calibration log

Both kinds matter. Engineers use the first constantly. The trouble starts when a kid uses the first to answer a question that needs the second.

What to Do at Home This Month

Make “how do you know?” a family reflex

When anyone in the house quotes a chatbot, someone asks, kindly, “How do you know that’s right?” The answer can be “I checked,” “it gave a source I opened,” or “I don’t know yet.” All three are fine. The habit is the point. Research on who kids trust online and why shows these norms form at home long before school.

Turn the confidence dial together

If your child uses a platform that lets kids build bots, sit in on one session and ask to see the settings. Point at the confidence or tone option and ask what it does. Then test it: same question, high confidence and low confidence. Watching the facts stay the same while the voice changes is a fast lesson.

Praise “I don’t know” from a bot

When a chatbot admits uncertainty or says it cannot verify something, point it out as a good sign, not a failure. Kids tend to read hedging as weakness. Reframe it as honesty.

Keep school-help bots to a higher standard

The study found kids already made school bots more formal and transparent than hobby bots. Build on that instinct: for homework, a bot’s answer is a lead to check, never the answer itself. Pew Research found in January 2025 that 26% of U.S. teens had used ChatGPT for schoolwork, double the 2023 share, and only 18% thought using it to write essays was acceptable. Teens have opinions here. Ask for theirs.

What not to do

Don’t assume a coding camp or chatbot-building class has handled this. Making a bot teaches how bots are put together. It does not, on its own, teach that a working bot can be a wrong bot. If the class never tested its bots against an answer key, that part is still yours.

What to Watch For Over the Next 3 Months

• Week 4: Does your child ever ask “how do you know?” about a chatbot answer without being prompted? Note when it happens.
• Month 2 red flags: quoting a chatbot as final authority in arguments (“ChatGPT said so”), turning in homework with unchecked AI facts, or calling a bot trustworthy because “it always answers.”
• Month 3 self-check: Ask your child to explain, in one sentence, the difference between a bot that works and a bot that’s right. If she can, and can give an example, the lesson took. If not, rerun Captain Sure.

The European Commission and OECD’s AI Literacy Framework lists “Create with AI” and “Manage AI” as separate domains. That split mirrors this article: making is one skill, judging is another. For a wider look at what kids need, see the three levels of AI literacy kids actually need.

Frequently Asked Questions

Why do kids trust chatbots so easily?

Chatbots answer instantly, fluently and in a confident tone, and children already lean toward machines for facts. A 2022 study found trust in a voice assistant for factual questions grew between ages 4–5 and 7–8. The 2026 chatbot-building study adds that ages 10 to 13 often equate trust with a bot doing its job, not with accuracy.

Does learning to build a chatbot make kids more skeptical?

Not automatically. In the Ozturk et al. preprint, some kids built bots that gave wrong answers on purpose and still called them trustworthy. Building teaches how bots work. Skepticism comes from testing what the bot says against answers you can check, then talking about the gap.

Is a chatbot’s confidence a sign it’s right?

No. A chatbot’s tone is shaped by its instructions and training, and it writes “definitely” with the same process it uses for facts. Common Sense Media’s 2026 testing of AI toys found they often gave incorrect answers while sounding confident. Treat tone as style, not evidence.

What age should kids start learning to question AI?

The groundwork starts early. By age 4, children prefer accurate speakers over inaccurate ones (Koenig & Harris, 2005), so screen-free games like Two Puppets work for 5- to 8-year-olds. Hands-on bot testing fits ages 9 to 12, and calibration logs suit teens.

Is this study reliable?

It is a preprint, not yet peer-reviewed, with 115 participants from the researchers’ learning programs. The age difference in confidence settings was statistically significant, but the abstract does not say how many kids made deliberately wrong bots. Treat it as a strong lead that fits older peer-reviewed research.

---

About the author

Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. Ozturk, D., Li, J., Singh, D. P., Rajapaksha, Y., Melese, F., Riahi, B., Jiang, S., Jin, Q., Huang, J., Cateté, V., Barnes, T., & Tian, X. (2026). “How Children Design and Reason about Trustworthy AI Chatbots.” arXiv (preprint, not peer-reviewed). https://arxiv.org/abs/2609.25244
2. Koenig, M. A., & Harris, P. L. (2005). “Preschoolers mistrust ignorant and inaccurate speakers.” Child Development, 76(6). https://doi.org/10.1111/j.1467-8624.2005.00849.x
3. Girouard-Hallam, L. N., & Danovitch, J. H. (2022). “Children’s trust in and learning from voice assistants.” Developmental Psychology. https://doi.org/10.1037/dev0001318
4. Common Sense Media. (2026, updated January 22). “AI Toys: Risk Assessment.” https://institute.commonsensemedia.org/risk-assessments/ai-toys
5. Pew Research Center. (2025, January 15). “About a quarter of U.S. teens have used ChatGPT for schoolwork, double the share in 2023.” https://www.pewresearch.org/short-reads/2025/01/15/about-a-quarter-of-us-teens-have-used-chatgpt-for-schoolwork-double-the-share-in-2023/
6. European Commission & OECD. (2026). “AI Literacy Framework (AILit).” https://ailiteracyframework.org/