How Language Models Generate Text: The Next-Word Prediction Machine

Here’s something worth trying at home. Open ChatGPT or any AI chatbot. Type: “Continue this sentence: The capital of France is…” It will say Paris. Now type: “Continue this sentence: The most delicious food in France is…” It will say something about croissants, or coq au vin, or cheese. That answer is also “correct” — but notice what changed. The first sentence has one true answer. The second has infinitely many valid completions.

ChatGPT doesn’t distinguish between them in any meaningful way. In both cases, it’s doing the same thing: predicting what word comes next based on patterns it saw during training.

That’s not an insult. It’s a description. And it’s one of the most clarifying mental models a kid can have about AI.

The Core Misunderstanding: “It Knows Things”

The phrase “ChatGPT knows that…” shows up constantly. In classrooms, in news articles, in parent conversations. And it’s subtly wrong in a way that matters.

Language models don’t “know” things the way you know your home address. They encode statistical patterns across an enormous training dataset. When those patterns reliably produce accurate outputs, we interpret that as knowledge. When they don’t — when the model confabulates or contradicts itself — we call it a hallucination. Both outputs come from the same mechanism.

The mechanism is: next-token prediction.

Every word you read from a language model was predicted one token at a time. The model took everything before the current position — your prompt, plus every word it had already generated — and asked: “Given all of this, what word (token) most probably comes next?” It sampled from a probability distribution, picked a word, added it to the sequence, and repeated.

That’s it. That’s the magic.

Explained Like You’re 5: The World’s Most Practiced Sentence-Finisher

Imagine someone who has read every book, article, website, and forum post ever written — billions of documents. They spent years doing nothing but reading. Now you give them a sentence fragment, and they finish it. They’re not making things up, exactly — they’re doing an extremely sophisticated version of “what usually comes after this.”

They’re brilliant at sounding like they know everything. But if you ask them something that wasn’t in the texts they read — or something that requires reasoning beyond pattern completion — they’ll still produce a fluent, confident sentence. It just might be wrong.

The difference between that person and ChatGPT is that ChatGPT does this billions of times per day, never gets tired, and doesn’t know the difference between the questions it can answer accurately and the ones it can’t.

How It Actually Works: Temperature and Sampling

Beyond simple next-word prediction, there’s a dial called temperature that controls how “creative” or “random” the model’s outputs are.

At low temperature (near 0), the model almost always picks the highest-probability next word. The output is predictable, repetitive, and safe. Ask it to complete “The sky is…” at low temperature and you’ll get “blue” every time.

At high temperature (near 1 or above), the model samples more broadly from the probability distribution — sometimes picking lower-probability words. The output gets more varied and surprising. Sometimes more creative. Sometimes nonsensical.

This is why AI writing assistants have settings like “more creative” vs. “more precise.” They’re literally adjusting the temperature parameter.

Another mechanism is top-k sampling and top-p (nucleus) sampling — instead of considering the full vocabulary of 50,000+ possible next words, the model limits itself to the top-k most likely options, or the smallest set of options whose probabilities sum to p. This keeps outputs coherent while still allowing variety.

None of this involves thinking. It’s probability distributions and sampling strategies applied to an enormous set of learned weights.

Why Kids Should Know This Today

A 2023 study in Computers & Education found that students who understood how language models generate text were significantly more likely to verify AI-generated content before using it academically, compared to students who had no model of how AI worked. Understanding the mechanism changed the behavior.

This matters beyond academic honesty. A child who knows AI is predicting the next word — not retrieving facts — is a child who asks better questions, catches errors faster, and uses AI outputs as drafts rather than final answers.

The National Education Association’s 2025 AI in Schools guidance explicitly recommends teaching students the “underlying mechanics” of AI tools, noting that conceptual understanding is more protective against misuse than policy-based restrictions alone.

For older kids, this also opens a window into careers. Prompt engineering — the practice of crafting inputs that reliably produce useful outputs — is partly an art, partly a science grounded in understanding how sampling and attention mechanisms work. It’s a real skill. And it starts with understanding that you’re guiding a statistical prediction machine, not asking a question of a knowledgeable entity.

How to Teach Your Kid About This

Ages 5–8: Finish the Sentence Game (Mad Libs Edition)

Write sentence fragments on strips of paper and ask your child to finish them as many different ways as they can. “The dog ran to the…” — yard? park? moon? Then pick the most common answer across 10 strips. Explain: “That most common answer is what a computer would pick if you turned the dial to ‘safe.’ A funny or surprising answer is what it picks if you turn the dial to ‘creative.’”

This gives kids a physical intuition for temperature before they ever touch a computer.

Ages 9–12: Compare Temperature Settings

Most AI chatbots don’t expose temperature directly, but you can mimic it. Ask the same creative question three times in a row: “Write me a one-sentence story about a robot.” Compare the three answers. Some will be similar; some will diverge. Then ask it the same factual question three times: “What is the boiling point of water?” The answers should be almost identical — because the high-probability token is “100 degrees Celsius” and the model almost always picks it.

The contrast illustrates how the same system handles creative vs. factual tasks differently — and why creative outputs vary while factual outputs are more consistent.

Ages 13+: Read About Sampling Strategies

The original transformer paper (Vaswani et al., 2017) and the GPT-3 paper (Brown et al., 2020) are freely available and surprisingly readable in their introductory sections. For a teenager, Andrej Karpathy’s free Neural Networks: Zero to Hero video series walks through building a character-level language model from scratch in Python. Watching text generate one character at a time makes next-token prediction viscerally clear.

Also worth exploring: how neural networks learn — the foundational mechanism underneath everything a language model does.

Temperature and Output: What Changes

Temperature Setting	Output Behavior	Best for	Downside
Very low (0–0.2)	Near-deterministic, always picks top word	Factual Q&A, code, structured data	Repetitive, robotic tone
Low-medium (0.3–0.6)	Mostly predictable, slight variation	Summaries, formal writing	Still constrained creatively
Medium (0.7–0.9)	Balanced variation, conversational	General writing, chat, brainstorming	Occasional non-sequiturs
High (1.0–1.2)	High variety, surprising outputs	Creative writing, poetry, ideation	Can veer into incoherence
Very high (1.5+)	Highly random, unpredictable	Experimental generation	Frequently produces nonsense

Most commercial chatbots run around 0.7–0.9 by default — balanced enough to be natural, constrained enough to stay coherent.

Real-World Examples Kids Encounter Every Day

Autocomplete on Gmail — “Smart Compose” finishes your sentences while you type. It’s a smaller, faster language model doing exactly next-token prediction. The reason it sometimes suggests something slightly off is that it’s predicting probability, not intent.

GitHub Copilot — used by professional developers to autocomplete code. Same mechanism: given the code written so far, what line probably comes next? It’s trained on billions of lines of public code, so its “next token” guesses are often correct.

AI essay feedback tools — when an AI tool gives feedback on a student’s essay, it’s predicting what feedback text usually follows an essay like this one. That feedback is statistically plausible — it might be genuinely helpful, or it might be generic noise that sounds like feedback.

Voice assistants — when Siri or Google Assistant responds to a question, there’s often a language model generating the response text (separate from the speech recognition that heard you). Same prediction mechanism, just with audio transcription on the front end.

What to Watch for Over 3 Months

Month 1: Does your child understand that ChatGPT generates text word by word? The simplest test: ask them “How does AI decide what word to write next?” If they say something like “it picks the most likely word based on everything before it,” the core concept is there.

Month 2: When using AI for school, does your child treat AI output as a first draft requiring verification rather than a final answer? The shift in behavior — from copy-paste to read-and-check — is the real payoff of this understanding.

Month 3: Can your child explain why AI sometimes gives different answers to the same question? “Because it’s sampling from a probability distribution — there’s randomness built in” is the accurate answer. If they can say that, they’re thinking at a level most adults don’t reach.

Red flag: if your child still refers to AI “knowing” things by month 3, revisit the sentence-finishing game. The insight usually comes from seeing variability — when the same question gets different answers, it’s obvious the model isn’t looking up a stored fact.

FAQ

Is ChatGPT just autocomplete? That seems too simple.

It’s a more complex version of autocomplete — the scale difference is enormous, and the architecture (transformer attention mechanisms) is sophisticated. But yes, the fundamental operation is next-token prediction. Calling it “just autocomplete” undersells the engineering; calling it “thinking” or “understanding” overstates what’s happening.

Why does ChatGPT sound so confident even when it’s wrong?

Because confidence is a stylistic pattern in its training data. Most authoritative text is written confidently. The model learned that confident tone predicts well — it shows up often in correct, helpful writing. So it applies that tone regardless of whether the underlying prediction is accurate.

Can I make ChatGPT be less creative or more creative?

Some interfaces expose this. In the ChatGPT app, you can’t directly set temperature, but choosing different modes (like GPT-4 vs. GPT-4o) affects behavior. In many API integrations and developer tools, temperature is directly adjustable. Asking the model to “be precise and factual” in your prompt also nudges it toward lower-temperature outputs — you’re influencing what high-probability completions look like.

Is generating text in Spanish harder for AI than English?

Yes, generally — because most large language models are trained on datasets that are heavily English-dominated. Spanish is usually well-represented (it’s the fourth most common internet language), but less common languages or regional dialects have much sparser representation in training data. This affects output quality directly.

Does the AI read my question all at once, or word by word?

It processes the full prompt (context window) at once using attention mechanisms — it’s not reading left to right sequentially in the way humans do. But it generates the response one token at a time, left to right. So processing your input is parallel; generating the output is sequential.

Should I be worried about my kid’s writing skills if they use AI?

Research is genuinely mixed on this. A 2024 study in Educational Psychology Review found that students who used AI for drafting but revised and rewrote showed similar skill development to those who didn’t use AI. Students who used AI as a substitute without engaging the output showed degraded writing performance. The difference is whether the child is thinking or just accepting. That’s a parenting conversation more than a technology question.

---

About the author Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
2. Brown, T., Mann, B., Ryder, N., et al. (2020). “Language Models are Few-Shot Learners.” Advances in Neural Information Processing Systems, 33. https://arxiv.org/abs/2005.14165
3. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). “On the Dangers of Stochastic Parrots.” Proceedings of FAccT 2021. https://dl.acm.org/doi/10.1145/3442188.3445922
4. National Education Association. (2025). AI in Schools: Guidance for Educators and Families. https://www.nea.org/professional-excellence/student-engagement/tools-tips/artificial-intelligence-schools
5. Ouyang, L., Wu, J., Jiang, X., et al. (2022). “Training Language Models to Follow Instructions with Human Feedback.” Advances in Neural Information Processing Systems, 35. https://arxiv.org/abs/2203.02155
6. Holzinger, A., Saranti, A., Molnar, C., Biecek, P., & Samek, W. (2022). “Explainable AI Methods.” Proceedings of AI+X Symposium. https://doi.org/10.1007/978-3-031-04083-2_16