Uploading Kids' Photos to AI Apps: What to Know First

In October 2026, a South Carolina mother learned that her 6-year-old’s school had used AI to turn the girl’s photo into an image of her as a 25-year-old nurse. Nobody had asked. The story spread fast because it touched something many parents feel but rarely say out loud: uploading kids’ photos to AI apps feels like harmless fun, yet once the face leaves your phone you no longer decide where it goes. It may be stored, used to train a model, copied into a dataset, or reused by someone you will never meet. None of that is guaranteed. All of it has happened. Here is what is known, and a four-question check to run before the next cartoon filter or “see your future self” trend.

Key Takeaways

• Once uploaded, a child’s photo is governed by the app’s terms and servers, not your camera roll. Look for three things: retention, training use and sharing.
• Human Rights Watch found 170 identifiable photos of Brazilian children in LAION-5B, a dataset used to train popular image tools, after reviewing less than 0.0001% of it. Many came from family blogs.
• Mainstream tools can be misused. In January 2026 the European Commission opened a Digital Services Act investigation into X after its Grok AI was used to make sexualized images of women and children.
• Law is starting to catch up. Spain’s cabinet approved a draft bill in January 2026 setting 16 as the age to consent to use of your own image. In the US, COPPA treats a photo of a child under 13 as personal information.
• The safest swaps are simple: use a drawing, an avatar or a photo with no face; ask your child first; and skip apps that claim rights to train on your uploads.

What Happened in South Carolina, and Why It Hit a Nerve

The case is logged in the OECD’s AI Incidents Monitor, dated October 20, 2026. Mother Nikki Anderson objected after her daughter’s school used AI to generate an image of the 6-year-old as a grown-up nurse, without parental consent. People, Yahoo News, the Economic Times and others covered it.

The OECD classifies it as an AI “hazard,” not an “incident.” That distinction is useful. A hazard means a situation that could plausibly lead to harm; no actual harm or rights violation has been confirmed, and the school’s response was pending when it was logged. Nobody is alleging the image was misused.

So why the reaction? Two separate questions got tangled together. One is consent: who decided that this child’s face could be fed into an AI tool? The other is authorship: who gets to imagine a child’s future? A drawing the girl made of herself as a nurse would have been hers. The AI version was the school’s, built from her face. Both questions apply to the apps parents use at home.

Uploading Kids’ Photos to AI Apps: What Happens to the File

Here is the mechanism, without the marketing. When you tap “upload,” the image file travels to the company’s servers. The app usually runs face detection to find and crop the face, then converts it into a set of numbers that capture its features, sometimes called an embedding. A generative model uses the image and those numbers to produce the stylized result, which is sent back to you.

What happens next depends on choices the company made and wrote into its terms:

• Retention. Is the original deleted after processing, after 30 days, or kept indefinitely?
• Training. Do the terms grant the company a license to use uploads to improve or train its models? Once a photo has influenced a model’s weights, it cannot be cleanly removed.
• Sharing. Are images shared with vendors, partners or “affiliates”?
• Public by default. Some apps post results to a public feed or gallery unless you turn it off.

Separately, anything posted publicly online, including the cartoon you share afterward, can be scraped. That is how the LAION-5B problem happened.

The dataset problem: photos that were never meant to be found

In June 2024, Human Rights Watch reported that LAION-5B, a dataset of about 5.85 billion images and captions scraped from the web and used to train popular AI tools, contained links to identifiable photos of Brazilian children. Researchers reviewed less than 0.0001% of the dataset and still found 170 photos of children from at least 10 Brazilian states, which they called likely a significant undercount. Some captions or URLs included children’s names; many showed where and when the photo was taken. Many had been posted by parents on personal blogs, some a decade before the dataset existed, and were not otherwise findable by a search.

LAION confirmed the photos were there and pledged to remove them. It disputed that models trained on the data could reproduce personal information verbatim, and said children and guardians are responsible for removing photos from the internet. Human Rights Watch argued that training on photos of real children has made it easier to create convincing fakes of any child.

Both sides have a point worth taking home. A trained model does not usually spit out your exact photo. But the photo did not need to be “reproduced” to have been used without anyone’s permission.

The misuse problem: it’s not only fringe apps

Parents often assume deepfake abuse comes from obscure “nudify” sites. In January 2026, the European Commission opened a formal investigation under the Digital Services Act into whether X addressed the risks after its Grok AI was used to create sexualized images of women and children from user prompts. The Centre for Countering Digital Hate estimated that Grok produced about 3 million sexualized images of women and children in a matter of days. Commission President Ursula von der Leyen said Europe would not tolerate the “digital undressing of women and children.”

The lesson for families is not panic. It is that a general-purpose AI built into a social app can be turned against a real person’s photo, so the fewer clear, public images of your child, the less raw material exists. For a full guide on prevention and response, see this parent’s guide to AI-generated fake images of kids.

What the Law Says About Children’s Faces

United States. Under the Children’s Online Privacy Protection Act, the Federal Trade Commission lists “a photograph, video, or audio file, where such file contains a child’s image or voice” as personal information. Apps directed at children under 13, or that knowingly collect from them, need verifiable parental consent. The rule was amended in April 2025. COPPA does not stop a parent from uploading their own child’s photo to a general-audience app.

Spain. On January 13, 2026, Spain’s cabinet approved a draft bill that sets 16 as the minimum age to consent to use of one’s own image, limits reuse of online images and AI-generated voices or likenesses without permission, and treats commercial use of AI-generated images or voices without consent as illegitimate. It still has to pass consultations and parliament.

European Union. The Digital Services Act gives the Commission power to investigate very large platforms over systemic risks such as illegal content. The X proceedings are ongoing; no final decision had been reported as of November 1.

If something has already gone wrong, the National Center for Missing & Exploited Children’s free Take It Down service helps remove explicit images taken of someone before age 18 from participating platforms. It works by creating a digital fingerprint (a hash) on your device; the image itself is not uploaded. Never download, screenshot or forward an explicit image of a minor to “save evidence”; that can itself be a crime. Report to the platform, to NCMEC’s CyberTipline and to police instead.

How to Teach Your Kid About Their Own Photos

Ages 5–8: the thumbs-up rule

Before you post or upload any photo of your child, show it to them and ask, “Is it OK if I share this?” Respect a thumbs-down. That is all. It builds the idea that a picture of you belongs partly to you, and it costs ten seconds. Extend it to friends: “We ask before we take a picture of someone else, too.”

Ages 9–12: the privacy-policy scavenger hunt

Pick an AI photo or avatar app your child is curious about. Open its privacy policy together and hunt for four words or ideas: “retain” (how long), “train” or “improve our services” (used for AI), “share” or “third parties” (who else), and “delete” (can you remove it). Give a point for each one you find and explain. Then decide together: does this app pass? Kids usually decide faster and more strictly than adults expect.

Ages 13+: audit your own face online

Have your teen search their full name in an image search and look at what comes up. Then check each of their accounts: which photos are public, which are tagged by others, which show a school logo or neighborhood. Make a plan to lock down or remove the most identifying ones. Discuss the consent side: when they post a group photo, did everyone in it agree? Teens who audit themselves tend to post differently afterward.

The question to ask: “If you upload a photo to an app, who decides what happens to it next, and how would you find out?”

Real Face vs. Safer Options: A Quick Comparison

Option	What leaves your phone	Training or reuse risk	Deepfake raw material	Good for
Clear photo of your child’s face	Full-resolution face, often with metadata	Depends on terms; may be retained or used to train	High	Avoid for novelty apps
Photo with face turned away or obscured	Body, setting, clothing	Lower	Low	Fun edits without identity
Child’s own drawing, scanned	A drawing	Very low personal risk	None	”Turn my drawing into…” projects
Built-in avatar or character creator	No photo at all	None from the face	None	Profile pictures, games
On-device editing (no upload)	Nothing, if processing truly stays local	None from the app	None from the app	Basic filters; check settings

Remember that the result you share also counts. A stylized cartoon still shows your child’s features, and once posted publicly it can be copied.

What to Do at Home Before the Next Photo Trend

Run the four-question check

Before any upload: (1) How long does the app keep the photo? (2) Can it use the photo to train AI? (3) Who does it share with? (4) Can I delete it, and how? If the app does not answer clearly, treat the answer as “forever, yes, others, no.”

Make your child’s consent the house rule

Ask before uploading, from age 5 up. It sounds small. It teaches children that their image is theirs, which is exactly the norm the South Carolina mother said was skipped. The research on sharenting and what posting does to kids’ privacy shows these habits shape how kids treat their own and others’ photos later.

Ask your school what tools it uses

Schools increasingly run student work and photos through AI tools. You can ask in writing: Which AI tools process student images? Is parental consent requested? Are images used for training? How long are they kept? A polite email often surfaces policies nobody has looked at in a while.

Prefer drawings and avatars

Most of the fun of AI image play survives without a real face. Let your child draw themselves as a future scientist, then use the drawing. It also hands the imagining back to the kid. For more on consent and image generators, see what parents should know about AI image generators and children’s photos.

What not to do

Don’t upload clear, front-facing photos of your child to apps whose terms you have not read, especially “free” ones. Don’t post the AI result publicly with your child’s name, school or location. And don’t assume deleting the app deletes the photo; deletion usually has to be requested separately.

What to Watch For Over the Next 3 Months

• Week 4: Go through the AI apps on family phones. Delete uploads where possible, turn off any “use my content to improve” setting, and remove apps that fail the four-question check.
• Month 2 red flags: your child uploading friends’ photos without asking, a new viral face-filter trend at school, or your child’s photo appearing somewhere you did not post it. Ask calmly how it got there.
• Month 3 self-check: Search your child’s name and your own social accounts for public, identifiable images. Can your child explain why you ask before posting?

Frequently Asked Questions

Is it safe to upload my child’s photo to an AI app?

It depends on the app’s terms. Some delete photos quickly and never train on them; others keep uploads and claim broad rights. Check retention, training use, sharing and deletion before uploading. If the policy is unclear, use a drawing or a photo without a face instead.

Can AI apps use my kid’s photo to train their models?

Some can, if their terms grant a license to use uploads to improve services. Separately, photos posted publicly can be scraped into training datasets, as Human Rights Watch documented with LAION-5B in 2024. Once a photo has shaped a trained model, it cannot be cleanly removed.

Can someone make a deepfake from one photo of my child?

Human Rights Watch says AI models can create convincing fakes from very few images, sometimes one. The EU’s 2026 investigation into X’s Grok shows mainstream tools can be misused this way. Fewer clear, public photos means less raw material. If it happens, use NCMEC’s Take It Down and report to police.

What should I do if my child’s school used AI on their photo without asking?

Ask the school in writing which tool was used, whether the image was stored or used for training, and whether it can be deleted. Ask for the policy on parental consent going forward. The South Carolina case was logged by the OECD as a hazard, not proven harm, which is a fair starting tone.

Does COPPA protect my child’s photos?

Partly. COPPA treats a photo containing a child’s image as personal information and requires parental consent for apps aimed at children under 13. It does not stop parents from uploading photos to general-audience apps, and it does not cover teens 13 and older.

---

About the author

Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. OECD AI Incidents Monitor. (2026, October 20). Incident record 2026-10-20-3a14: school AI image of a 6-year-old made without parental consent (classified as an AI hazard). https://oecd.ai/en/incidents/2026-10-20-3a14
2. Human Rights Watch. (2024, June 10). “Brazil: Children’s Personal Photos Misused to Power AI Tools.” https://www.hrw.org/news/2024/06/10/brazil-childrens-personal-photos-misused-power-ai-tools
3. Al Jazeera. (2026, January 26). “EU probes Musk’s Grok AI feature over deepfakes of women, minors.” https://www.aljazeera.com/news/2026/1/26/eu-launches-probe-into-grok-ai-feature-creating-deepfakes-of-women-minors
4. Majorca Daily Bulletin. (2026, January 13). “Spain moves to curb AI deepfakes, tighten consent rules on images and teen social media use.” https://amp.majorcadailybulletin.com/news/local/2026/01/13/139531/spain-moves-curb-deepfakes-tighten-consent-rules-images-and-teen-social-media-use.html
5. Federal Trade Commission. “Complying with COPPA: Frequently Asked Questions.” https://www.ftc.gov/business-guidance/resources/complying-coppa-frequently-asked-questions
6. National Center for Missing & Exploited Children. “Take It Down.” https://takeitdown.ncmec.org/