Synthetic Data: How AI Is Training Itself on Data It Generated

Here’s a fact that should make you pause: some of the AI models your kids interact with were partly trained on text or data generated by previous AI models. Not deliberately, necessarily — much of the internet is now AI-generated content, and models trained on internet scrapes pick it up whether they intend to or not. But some labs are also doing it deliberately, using one model to generate training data for another.

This is the synthetic data story. Parts of it are practical and useful. Parts of it are a warning about what happens when AI becomes its own primary source.

Key Takeaways

• Synthetic data — data generated artificially rather than collected from real-world events — is used to train AI in medical imaging, autonomous driving, and privacy-sensitive domains where real data is scarce or regulated.
• Model collapse is a documented failure mode where models trained on AI-generated data produce increasingly distorted outputs, as errors compound across generations.
• High-quality curated synthetic data, used deliberately for specific gaps in real data, works well. Training on unfiltered AI-generated internet content is different and riskier.
• Data quality matters more than data quantity — a consistent finding across synthetic data research.
• Understanding the data origin of AI systems is part of being an informed AI user.

Why Synthetic Data Exists in the First Place

Real data is expensive, rare, private, or biased. Synthetic data solves each of those problems differently.

Medical imaging is the clearest example. Training an AI to detect tumors in CT scans requires thousands of labeled examples — expert radiologists identifying and marking every suspicious region. That labeling is expensive and slow. The real scans contain protected health information. And rare conditions, by definition, appear infrequently in any one hospital’s records.

Researchers at Stanford, MIT, and in industry have responded by generating synthetic CT scans — realistic images created by generative AI models trained on the real thing. A diffusion model trained on de-identified scans can produce novel synthetic scans with anatomically plausible features, including synthetic tumors of specified types and sizes. AI trained partly on this synthetic data achieves performance comparable to AI trained on real data alone, while training on more examples without additional privacy risk (Frid-Adar et al., 2018, Neurocomputing).

Autonomous driving faces a data problem with rare events. A self-driving car’s AI must handle thousands of scenarios — a child running into the street, a motorcycle splitting lanes in rain, a sudden highway accident — that each appear rarely in normal driving. Collecting enough real-world examples of every rare scenario to train on would require millions of miles driven. Game engines and physics simulators (CARLA, NVIDIA DRIVE Sim) generate synthetic driving data for precisely these edge cases. Waymo, Tesla, and others use significant volumes of synthetic driving data in their training pipelines.

Privacy preservation is a third motivation. Federated learning (where models train on data that never leaves users’ devices) is one approach. Synthetic data is another: generate a synthetic dataset that has the same statistical properties as sensitive real data, then share only the synthetic version. Differential privacy guarantees can be incorporated into the generation process, providing mathematical bounds on how much the synthetic data could reveal about any individual in the original set.

What Model Collapse Is and Why It Matters

In 2023, a team at Oxford published a paper in Nature (Shumailov et al., 2024) describing what they called model collapse: when a model is fine-tuned on outputs from a previous generation of the same or similar model, performance degrades. Train a second-generation model on first-generation outputs. Train a third-generation model on second-generation outputs. By generation five or six, the model’s output distribution has converged to a narrow, distorted version of the original — heavy tails of the data distribution (rare words, rare concepts, edge cases) get progressively erased.

Think of it like photocopying a photocopy. Each generation adds noise and loses detail at the edges. What’s left is a blurry approximation of the original.

The model collapse concern is most acute for language models, where AI-generated text has proliferated across the internet. If future language models are trained on web scrapes that contain substantial proportions of AI-generated text (blog posts, news articles, social media content), the training data is increasingly “second-generation” — a copy of a copy. Research from MIT and other groups is now studying how to detect and filter AI-generated content in training corpora specifically to prevent this.

Synthetic Data Use Case	Quality Assessment	Model Collapse Risk
Medical image augmentation (curated, labeled)	High — often improves model performance	Low — specific, controlled application
Autonomous driving rare-event simulation	High — physics-simulated, realistic	Low — physics constraints provide ground truth
Language model fine-tuning on AI-generated text (deliberately curated)	Medium — depends on quality filtering	Medium — requires careful filtering
Language model trained on unfiltered internet (contains AI-generated content)	Low — uncontrolled AI content mixed in	High — compounding errors across generations
Privacy-preserving synthetic tabular data	Variable — depends on generation method	Low for structured data — different collapse dynamics

What Successful Synthetic Data Use Looks Like

The cases where synthetic data works well share common features:

Physical or biological constraints provide ground truth. Synthetic medical images must be anatomically plausible. Synthetic driving data must obey the physics of vehicle motion. These constraints prevent the worst runaway distortions.

The synthetic data fills a specific gap rather than replacing all real data. Best practice is synthetic data augmentation — using synthetic examples to supplement real data in underrepresented categories, not replacing real data wholesale.

Careful quality filtering removes low-quality synthetic examples before training. Unfiltered AI-generated text is the risky scenario. Curated, reviewed synthetic examples are different.

The generation model is better than (or at least as good as) the training target. Using a large, capable model to generate training data for a smaller, cheaper model (knowledge distillation) works well. Using a lower-quality model to generate data for a model you want to improve is how collapse starts.

How to Teach Your Kid About Synthetic Data

Ages 5–8: Draw your own pictures to teach a robot

Explain that AI learns to recognize cats by looking at millions of pictures of cats. Ask: “What if there aren’t enough real photos? Could you draw a cat instead?” Let them draw five cats. Then discuss: “Would a robot learn the same things from your drawings as from photos? What might it get wrong?” This introduces the concept that training data shapes what the model learns.

Ages 9–12: Play the telephone game with AI

The telephone game is a natural analogy for model collapse. Whisper a sentence; it transforms with each retelling. Do the experiment: have your child write a sentence, feed it to an AI that paraphrases it, feed the paraphrase back to the AI for another paraphrase, and repeat five times. Read the original and the fifth-generation version together. The drift illustrates what happens when AI output becomes the input for the next round of training.

Ages 13+: Study the model collapse paper

The Shumailov et al. (2024) Nature paper is available as an open-access preprint on arXiv. The core experiment — train models on synthetic data from previous generations, measure the output distribution’s collapse — is described clearly. A teenager interested in AI can read the abstract, methods, and figure 2 (which shows the distribution narrowing) and understand the core finding. Connect this to the broader question of how AI learns and where its knowledge actually comes from.

The question to ask: “If an AI was trained on data generated by another AI, which was trained on data from the internet, and the internet now has a lot of AI-generated content in it — how would you figure out where the AI’s ‘knowledge’ actually came from?”

What to Watch For Over the Next 3 Months

Month 1: Notice AI-generated content online. AI content detection tools (GPTZero, Originality.ai) are imperfect but can give a sense of the prevalence of AI-generated text in any given domain. The question “how much of this article was written by AI?” is increasingly relevant to evaluating online sources.

Month 2: When your child uses AI for research or learning, practice asking: “Where did this AI’s information come from? Was it trained on real sources or other AI outputs?” The AI won’t always know, but asking the question builds the habit of provenance thinking.

Month 3: If your teenager is interested in AI/ML, look at the Hugging Face datasets repository (huggingface.co/datasets) — it lists thousands of training datasets with documentation about their sources, including which ones contain AI-generated content. Understanding dataset composition is a real skill in ML engineering.

Frequently Asked Questions

Is all AI-generated training data bad?

No. Deliberately curated synthetic data — medical images generated by physics-accurate models, autonomous driving simulations, privacy-preserving synthetic tabular data — works well and solves real problems. The risk is unfiltered or low-quality AI-generated data, especially when used to train language models that then generate more internet content, which gets scraped into future training sets.

How do I know if an AI was trained on synthetic data?

For most commercial AI products, you don’t — training data composition is generally not disclosed. Some research models and open-source models publish detailed “data cards” documenting what they were trained on. The Meta LLaMA 3 data card and similar documents give some transparency into data sources. Expecting disclosure should become a normal consumer expectation.

Does synthetic medical data mean AI doctors will have imaginary experience?

This is the right concern, asked precisely. The safeguard is that synthetic medical data is used for augmentation, not replacement, and is validated against real clinical performance before deployment. An AI trained partly on synthetic tumor images must still demonstrate its tumor detection accuracy on real patient data in clinical trials. The FDA’s Software as a Medical Device (SaMD) framework requires clinical validation regardless of training data composition.

Can data quality really beat data quantity?

This has been one of the consistent findings in recent large language model research. Meta’s LLaMA 3 team and others have published evidence that smaller models trained on carefully curated, high-quality data can outperform larger models trained on noisier, larger datasets. This is partly why synthetic data, when carefully curated, can improve models — not because more data is better, but because more relevant, high-quality data is better.

---

About the author Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. Shumailov, I., et al. (2024). “AI Models Collapse When Trained on Recursively Generated Data.” Nature, 631, 755–759. https://doi.org/10.1038/s41586-024-07566-y
2. Frid-Adar, M., et al. (2018). “GAN-Based Synthetic Medical Image Augmentation for Increased CNN Performance in Liver Lesion Classification.” Neurocomputing, 321, 321–331. https://doi.org/10.1016/j.neucom.2018.09.013
3. Dosovitskiy, A., et al. (2017). “CARLA: An Open Urban Driving Simulator.” CoRL 2017. https://arxiv.org/abs/1711.03938
4. Jordon, J., et al. (2022). “Synthetic Data — What, Why and How?” Royal Statistical Society Series A. https://arxiv.org/abs/2205.03257
5. National Institute of Standards and Technology. (2023). “Synthetic Data for AI/ML Use Cases.” https://www.nist.gov/system/files/documents/2023/03/14/NIST.SP.1500-26.pdf
6. Wornow, M., et al. (2023). “The Shaky Foundations of Large Language Models for Healthcare.” npj Digital Medicine, 6, 135. https://doi.org/10.1038/s41746-023-00879-8