Table of Contents
AI Image Generators and Your Kids: Consent, Privacy, and What Can Go Wrong
What happens when your child's photos are used to train AI models or generate deepfakes—and how to protect image consent, opt out of training, and talk to kids about it.
A parent in Ohio posted birthday photos of her 11-year-old daughter to a popular AI avatar app. Three weeks later, her daughter’s face appeared in AI-generated content on a forum the parent had never heard of. The app’s terms of service included a clause granting the company “a worldwide, royalty-free license to use, reproduce, modify, and distribute any content you upload.”
This scenario is no longer rare. As AI image generation tools become easier to use and more powerful, the gap between what parents assume these platforms do with photos and what they actually do has widened into something parents urgently need to understand.
Key Takeaways
- Many AI image platforms claim broad rights to uploaded photos in their terms of service, including the right to use them for model training
- AI-generated deepfakes of minors are a growing problem—both for non-consensual content creation and for sextortion
- Legal protections for children in AI-generated imagery vary significantly by state; federal law is still catching up
- Teaching children what “image consent” means is now as important as teaching them about password safety
- Opting out of AI training on major platforms is possible but often buried in privacy settings
What AI Image Generators Actually Do With Photos
When a user uploads a photo to an AI image generation service, several things can happen depending on the platform’s data practices:
Image analysis for the immediate task. The photo is processed to complete the requested action—applying a style, generating an avatar, swapping a background.
Retention for model improvement. Many platforms retain uploaded images and use them to improve their AI models. This is typically disclosed in terms of service but rarely in the interface where the upload happens.
Third-party sharing. Some platforms share uploaded data with model training partners or subsidiaries. The photo of your child uploaded to App A may end up in a training dataset controlled by Company B.
No deletion guarantee. Even when users delete accounts, many platforms retain uploaded content for extended periods. “Deleted” doesn’t mean removed from training datasets already compiled.
The key variable is the platform’s terms of service and privacy policy. These documents are often 10,000+ words long, written in legal language, and change without prominent notification.
Platform Terms of Service: What to Look For
Before uploading any photo of a child to an AI platform, look for these specific clauses:
License grants: Phrases like “royalty-free, worldwide license to use your content” or “non-exclusive license to reproduce, distribute, and create derivative works” indicate the platform claims rights to the uploaded image beyond its immediate use.
Training data language: Look for phrases like “to improve our services,” “to train our models,” or “for product development.” These are signals that your uploads may be used for AI training.
Anonymization claims: Some platforms claim they anonymize or de-identify images before training. The quality of anonymization varies widely and is rarely auditable by users.
Opt-out provisions: Better-practice platforms include an opt-out from training data use. These are often buried in privacy settings or require contacting support directly.
| Platform Type | Typical Training Data Practice | Opt-Out Available |
|---|---|---|
| Major social media | User images used to improve features | Partial (varies by region) |
| AI avatar apps | Often explicit training use in ToS | Rare |
| Adobe Firefly | Does not train on user-uploaded content | N/A (not collected) |
| Consumer AI image tools | Varies widely; read ToS | Sometimes |
| Professional creative tools | Often cleaner data practices | More common |
AI-Generated Deepfakes Involving Children
The more serious concern isn’t what companies do with uploaded photos for training purposes—it’s what individuals do with AI image generation tools to create non-consensual imagery of minors.
Deepfake technology has reached a point where a handful of photos from a social media profile can be used to generate realistic, convincing images of a person in scenarios they never appeared in. For children, this has two primary threat vectors:
Non-consensual intimate imagery (NCII): AI-generated sexual content depicting real children (even when created synthetically, not from actual abuse) is illegal under federal law and the laws of most states. The FBI reported a significant increase in sextortion cases involving AI-generated imagery in 2023 and 2024. In some cases, attackers create this content and then threaten to distribute it unless the child provides money or real images.
Identity-based manipulation: AI-generated images of children can be used to create fake social media profiles, generate plausible-seeming evidence for false accusations, or manipulate other children in online communities.
For the specific threat of sextortion involving AI-generated imagery, our guide on sextortion and what parents need to know in 2026 covers response steps in detail.
Legal Status of AI-Generated Child Imagery
The legal landscape is evolving rapidly and unevenly.
Federal law: The PROTECT Act of 2003 and CIPA both address child sexual abuse material (CSAM) and extend to computer-generated imagery that depicts minors in sexually explicit contexts. AI-generated imagery depicting child sexual abuse is prosecutable under federal law regardless of whether a real child was involved in its creation.
State laws: As of 2026, 38 states have enacted or are actively developing laws specifically addressing AI-generated deepfakes. Some states (California, Texas, Virginia, New York) have enacted laws specifically targeting non-consensual deepfakes. Many of these laws include provisions for minors.
The enforcement gap: The legal framework exists, but enforcement is genuinely difficult. AI-generated content is created and distributed faster than law enforcement can track, platforms vary in their reporting compliance, and international jurisdictional issues complicate prosecution of content generated or hosted abroad.
What this means for parents: Legal protection exists but isn’t automatic. If your child is the subject of AI-generated harmful imagery, the reporting path runs through the National Center for Missing & Exploited Children (NCMEC) CyberTipline, the FBI’s IC3, and potentially your state attorney general’s office.
How Children’s Photos End Up in AI Training Datasets
The pathway isn’t always through direct uploads to AI platforms. Children’s photos reach AI training datasets through several channels most parents don’t consider:
Public social media. Photos posted publicly on Facebook, Instagram, and similar platforms have been harvested for AI training datasets. The LAION-5B dataset, which was used to train several major AI image models, has been found to contain images scraped from public social media posts.
School platforms. Educational platforms that request student photos (for ID cards, class pages, or learning management systems) may have data-sharing practices that aren’t obvious to parents. Review your child’s school’s privacy policy specifically for AI-related provisions.
Shared public photos. Photos uploaded to platforms like Flickr with public licensing have been included in training datasets. Photos you shared 10 years ago may already be in datasets powering current AI models.
Third-party app connections. When your child’s account on one platform connects to another (common with social logins), data—including photos—may flow between platforms under combined privacy policies.
How to Opt Out of AI Training on Major Platforms
Opting out is possible on most major platforms but requires knowing where to look:
Google: myaccount.google.com → Data & Privacy → “Data used to personalize Google services.” Some AI-specific opt-outs are located under individual product settings.
Meta (Facebook/Instagram): Settings → Privacy → Generative AI. Meta introduced an AI training opt-out in 2024 under regulatory pressure in the EU. The opt-out form is available globally but must be submitted through a separate request form, not a simple toggle.
X (Twitter): Settings → Privacy and Safety → Grok → uncheck “Allow your posts and interactions to train Grok AI models.”
Adobe (Creative Cloud): Adobe has stated that paid Creative Cloud subscribers’ personal content is not used to train Adobe’s AI models (Firefly). This applies to Lightroom, Photoshop, and other Creative Cloud apps.
AI avatar and filter apps: Most third-party AI avatar apps do not offer opt-outs. The most effective protection is not uploading photos of children to these services. If you have already done so and want removal, contact the platform’s privacy team directly—under GDPR (in Europe) and California’s CCPA, you have rights to request deletion of personal data.
Teaching Kids About Image Consent
Image consent is a concept kids can understand starting around age 9 or 10: just as you ask before touching someone, you ask before taking, sharing, or modifying someone’s photo.
The AI dimension adds a new layer: photos shared online don’t stay in the context they were shared in. A photo shared in a group chat can be screenshot and distributed. A photo shared on a profile can be used to train an AI. A photo sent to a friend can be used to create an edited version that the friend never anticipated.
Teach these questions before any photo is shared:
- Would I be okay if this photo was seen by anyone, anywhere, forever?
- Did the person in this photo agree to have their image shared this way?
- Could this photo be used to create something else?
Our article on teaching kids to recognize phishing and manipulation covers the broader skill of evaluating digital situations before acting—the same framework applies to image sharing.
The Conversation You Need to Have
Many teens are already using AI image tools—face swap apps, AI filters, avatar generators. The conversation isn’t about banning these tools. It’s about:
What they can do with photos: Explain that uploading photos to AI platforms may mean those photos are retained, used for training, and potentially used in ways you can’t predict or reverse.
What consent means for images: Creating an AI-modified image of someone without their knowledge—even a friend, even a “funny” edit—is a consent violation. This becomes a much more serious issue if the modification is sexual, embarrassing, or designed to deceive.
What to do if something goes wrong: If your child encounters AI-generated content depicting them or a peer, they should tell a parent immediately. Don’t try to handle it alone. Our guide on reporting cybercrime involving kids walks through who to contact and how.
What to Watch For Over the Next 3 Months
The summer period brings increased AI tool use among teens, particularly for creating avatars, meme images, and social media content. Watch for apps that suddenly ask for camera access or photo library access without clear explanation of why. Review what AI-adjacent apps your child has installed and what permissions they’ve been granted.
Legislative activity on AI-generated deepfakes is accelerating. Several states have pending legislation that would extend existing protection frameworks specifically to AI-generated images of minors. Follow your state’s technology legislation for updates.
Frequently Asked Questions
Is it safe to use an AI avatar app that’s highly rated with good reviews?
App store ratings don’t reflect data practices. High ratings measure user experience, not privacy protection. Before using any AI image app with photos of your child, search for “[app name] + privacy policy” and look specifically for training data language and license grants. If the app doesn’t clearly state it doesn’t use uploaded images for training, assume it does.
Can I remove my child’s photos from AI training datasets if they’re already there?
In most cases, no—not from datasets that have already been compiled and distributed. You can request deletion from specific platforms under GDPR or CCPA if you can identify that they hold your child’s data. For datasets already used to train deployed models, the images are effectively embedded in the model weights, which cannot practically be “untrained.” Prevention is significantly more effective than remediation.
My child’s school uses an AI platform that scans student photos. What are my rights?
Under COPPA, platforms serving children under 13 face strict requirements around data collection, including photos. FERPA protects student educational records including photographs. Request the school’s data processing agreement with the AI platform and look for provisions about training data use and data sharing with the vendor’s parent company. You have the right to request this information.
What’s the difference between an AI filter in a social media app and an AI image generator?
Both can involve uploading your photo to a server for processing, but the risk profile differs. Social media AI filters are typically processed and discarded quickly for performance reasons, though terms vary. Dedicated AI image generators more commonly retain and use uploaded images. The key is reading the specific platform’s terms rather than assuming based on category.
About the author Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.
Sources
- Federal Bureau of Investigation. “Sextortion Involving AI-Generated Images.” fbi.gov, 2024.
- National Center for Missing & Exploited Children. “CyberTipline Annual Report 2024.” missingkids.org.
- Federal Trade Commission. “Children’s Online Privacy Protection Act (COPPA).” ftc.gov.
- LAION Foundation. “LAION-5B Dataset Documentation.” laion.ai.
- Electronic Frontier Foundation. “AI and Privacy: What You Need to Know.” eff.org.
- California Legislature. “AB 602 — Depiction of Individual Using Digital or Electronic Technology.” legislature.ca.gov.
- PROTECT Act of 2003. 18 U.S.C. § 2256.
