Algorithmic Bias: What It Is, Why It Happens, and How to Talk to Kids About Unfair AI

Your kid’s school might already use an AI system to flag “at-risk” students. Colleges are experimenting with AI application reviewers. Courts in several U.S. states use AI tools to help set bail and predict recidivism. These systems sound objective because they involve numbers and computers. They are not. A landmark study found that one widely used criminal justice algorithm flagged Black defendants as future criminals at nearly twice the rate of white defendants — with similar prior records. The math was working perfectly. That was the problem.

Algorithmic bias is one of the most important concepts a parent can understand right now, because AI systems are making consequential decisions about children — in schools, in child welfare systems, in the college admissions pipeline — and most of those systems are invisible to families. This article explains what algorithmic bias actually is, where it comes from, what the research shows, and how to make it concrete for kids at different ages.

Key Takeaways

• Algorithmic bias happens when AI systems produce systematically unfair outcomes for certain groups — not because of a bug, but because the training data encodes historical human biases.
• The COMPAS criminal risk tool incorrectly flagged Black defendants as high-risk at roughly twice the rate of white defendants with similar records (ProPublica, 2016).
• MIT researcher Joy Buolamwini found commercial facial recognition systems had error rates up to 34.7% for darker-skinned women vs. 0.8% for lighter-skinned men (Gender Shades, 2018).
• A 2019 study in Science found a healthcare algorithm systematically underserved Black patients — not because of race as an input, but because it used healthcare costs as a proxy for health need.
• Children can be taught to spot and question algorithmic bias through hands-on sorting games, free online tools, and simple Python projects.

What Algorithmic Bias Actually Is (and Isn’t)

Here is the most common misconception: people hear “algorithmic bias” and imagine a programmer who deliberately wrote racist or sexist code. That almost never happens. Algorithmic bias is usually the opposite — it’s what occurs when you take a mathematical system that is perfectly neutral and train it on human data that is not.

Historical data encodes history. If a hiring algorithm is trained on 20 years of successful employee records from a company that hired mostly men, it will learn that “successful employee” correlates with being male — even if gender is never an explicit input. The algorithm isn’t wrong by its own logic. Its logic was built on a biased world.

There are three main sources of algorithmic bias:

Training data bias. The dataset reflects historical inequalities. Amazon famously scrapped an AI hiring tool in 2018 after discovering it penalized resumes that included the word “women’s” (as in “women’s chess club”) and downgraded graduates of all-women’s colleges. The model had learned from a decade of Amazon hiring decisions made predominantly by and for men.

Proxy variable bias. The algorithm doesn’t use a protected characteristic directly but uses a correlated variable that acts as a stand-in. The Science study on healthcare algorithms (Obermeyer et al., 2019) found that hospitals were using healthcare costs as a proxy for health need. Because of systemic barriers to care, Black patients with the same health conditions had historically spent less on healthcare — so the algorithm concluded they were healthier than comparable white patients, and gave them lower priority scores.

Feedback loop bias. Predictive policing algorithms direct more police to neighborhoods historically over-patrolled — which generates more arrests in those areas — which becomes data confirming the algorithm’s prediction. The bias amplifies itself.

The Research That Changed the Conversation

Three studies reshaped how the AI field thinks about bias. Your kids will likely encounter all three in high school or college.

The ProPublica COMPAS Investigation (2016). Investigative journalists at ProPublica analyzed the COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) risk assessment tool used in U.S. courts to help judges decide bail, sentencing, and parole. They found the tool assigned higher recidivism risk scores to Black defendants who did not re-offend, and lower risk scores to white defendants who did re-offend. The false positive rate for Black defendants was nearly double that for white defendants. The company that built COMPAS disputed the analysis, and the statistical debate that followed was genuinely complex — it turns out some definitions of “fairness” are mathematically incompatible with each other. But the core finding held: the algorithm’s outputs were not race-neutral.

The Gender Shades Study (Buolamwini & Gebru, 2018). MIT Media Lab researcher Joy Buolamwini was coding a facial recognition project when she noticed it kept failing to detect her face — a dark-skinned woman’s face. She investigated. She and researcher Timnit Gebru tested three major commercial facial recognition systems (IBM, Microsoft, Face++) against a carefully constructed dataset of faces that was balanced by gender and skin tone. The results were striking. Error rates for lighter-skinned males were as low as 0.8%. Error rates for darker-skinned females reached 34.7%. The models had been trained predominantly on lighter-skinned faces, and they performed accordingly.

The Healthcare Algorithm Study (Obermeyer et al., 2019). Published in Science, this study examined a widely deployed algorithm used to identify which patients needed extra medical support. The algorithm had a significant racial bias despite never explicitly using race as an input. As described above, it used healthcare costs as a proxy — a proxy that was itself shaped by decades of unequal access to care. The researchers estimated that correcting the algorithm’s bias would more than double the percentage of Black patients receiving extra care, from 17.7% to 46.5%.

Where Kids Already Encounter Algorithmic Bias

This isn’t an abstract future problem. Children interact with biased algorithms regularly.

Context	Where Bias Appears	Documented Example
Social media feeds	Recommendation algorithms amplify engagement over accuracy	Facebook internal research showed content moderation hit Black and Latino users disproportionately (WSJ, 2021)
Search engines	Autocomplete and image results reflect societal biases	Google image search for “CEO” returned overwhelmingly male images for years
AI grading tools	Writing evaluation tools trained on data from specific demographics	EdTech tools have shown score disparities by dialect (Racial Equity in AI, Stanford HAI, 2021)
Facial recognition	Used in school security systems	Lockport NY schools deployed facial recognition, later paused after ACLU civil rights concerns
Predictive risk tools	Child welfare systems score family risk	A study of child welfare algorithm in Allegheny County found racial disparities in screening decisions (Eubanks, 2018)

What Responsibility Looks Like (and Doesn’t)

Tech companies are not entirely passive here. Many have published AI fairness principles, released bias auditing tools, and hired researchers to address these issues. But there’s a meaningful gap between stated principles and operational reality.

Regulation is fragmented. The EU AI Act (2024) classifies high-risk AI applications — including those used in education, employment, and criminal justice — and requires bias audits and human oversight. The U.S. has no equivalent federal law, though several states (Illinois, New York, Colorado) have passed narrower measures targeting specific uses.

Buolamwini’s follow-up work, through her Algorithmic Justice League, has pushed companies toward what she calls “coded bias accountability.” After the Gender Shades study, Microsoft and IBM updated their systems and reported improved accuracy. But third-party verification of those improvements has been inconsistent.

The honest answer for parents: the burden of oversight is falling on regulators, researchers, and advocates because corporate self-regulation has proven insufficient.

How to Teach Your Kid About Algorithmic Bias

Ages 5–8: The Sorting Game

Set up a simple sorting game at the kitchen table. Use a deck of playing cards. Tell your child the “rule” is to sort cards into two piles: one for cards the computer “likes” and one for cards the computer “doesn’t like.” Start by secretly making the rule “put red cards in the like pile.” Have your child play several rounds.

Then ask: “Does this rule seem fair to black cards? What if the rule had been made by someone who just happened to see a lot of red cards first?” Point out that the rule isn’t random — it was shaped by whoever made it. That’s how training data works. The computer learns the rule from examples, and if the examples favor one group, the rule will too.

The question to ask: “Who decided the rule, and what did they see when they made it?”

Ages 9–12: Find the Bias in Real Tools

Look up the Gender Shades study results together at gendershades.org (the site shows the accuracy gaps visually). Then, if you have a home computer with a webcam, try one of the free facial analysis demos available from researchers at MIT. Notice whether the tool performs differently depending on lighting, skin tone, or glasses.

You can also search for “Amazon AI hiring bias” or “COMPAS recidivism algorithm” and read a news article together. Ask: “What data do you think they used to train this? What was missing?”

The question to ask: “What group of people do you think made up most of the training data for this tool?”

Ages 13+: Train a Biased Classifier

Use Google’s Teachable Machine (teachablemachine.withgoogle.com) to build a simple image classifier. Here’s the experiment: train it using only images from a narrow category — for example, only photos of apples on white backgrounds — then test it with apples on wooden cutting boards, green apples, or small crabapples. Watch it fail. Discuss why.

For more advanced students: use a simple Python script (or a Jupyter notebook in Google Colab) to train a basic text classifier on a deliberately skewed dataset. Observe how accuracy differs across groups. This is the hands-on version of what Obermeyer’s team did with healthcare data.

The question to ask: “If you were building a hiring algorithm, what data would you collect, and what biases might already be in that data before you start?”

What to Watch For Over 3 Months

If your child is learning about AI in school, watch for a few things:

Month 1: Does the curriculum distinguish between “AI is wrong” and “AI is producing biased outcomes”? The distinction matters. A child who conflates bugs with bias will misunderstand how to fix either.

Month 2: Is your child asking “who built this, and using what data?” about technology they encounter? That question — more than any specific fact about bias — is the habit of mind that will serve them long-term.

Month 3: If your teenager is interested in tech careers, point them toward the field of algorithmic auditing. The demand for people who can independently evaluate AI systems for bias is growing faster than the supply. MIT, Stanford, and Carnegie Mellon all have active research programs here.

Red flag to watch: if your child concludes that AI is hopelessly broken and can’t be trusted — that’s overcorrection. The goal is calibrated skepticism, not blanket rejection. Good AI systems exist. The skill is knowing which questions to ask.

Frequently Asked Questions

Is algorithmic bias illegal?

In the U.S., it depends on the context. If an AI system produces discriminatory outcomes in employment, housing, or lending, it may violate existing anti-discrimination laws like the Fair Housing Act or Title VII — even if the discrimination is unintentional. The EU AI Act creates more explicit requirements for high-risk AI systems. But enforcement is inconsistent, and proving AI discrimination legally is difficult.

Can you just remove race and gender from the data to fix bias?

Removing protected characteristics often doesn’t work because of proxy variables — other data points that are statistically correlated with race or gender. Address, education level, and healthcare spending all contain racial signal. Some researchers call this the “fairness through blindness” fallacy. Real debiasing requires understanding the correlation structure of your data, not just deleting columns.

Do kids encounter biased AI at school?

Increasingly, yes. AI tools for grading essays, flagging at-risk students, monitoring test-taking, and recommending academic pathways are deployed in many U.S. districts. Most schools don’t publish the algorithmic systems they use or their accuracy data. Parents can request this information under FERPA in many cases.

How is this different from just human bias?

Both matter, but scale changes the stakes. A single biased hiring manager affects one hiring decision at a time. A biased algorithm deployed across thousands of schools, courts, or hospitals applies the same bias simultaneously to millions of people — and often without any human reviewing the individual decisions. The speed and scale of automation make systematic bias harder to catch and more consequential.

---

About the author

Ricky Flores is the founder of HiWave Makers and an electrical engineer with 15+ years of experience building consumer technology at Apple, Samsung, and Texas Instruments. He writes about how kids learn to build, think, and create in a tech-saturated world. Read more at hiwavemakers.com.

---

Sources

1. Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). “Machine Bias.” ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing
2. Buolamwini, J., & Gebru, T. (2018). “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification.” Proceedings of Machine Learning Research, 81, 1–15. http://proceedings.mlr.press/v81/buolamwini18a.html
3. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). “Dissecting Racial Bias in an Algorithm Used to Manage the Health of Populations.” Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
4. Dastin, J. (2018). “Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women.” Reuters. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
5. Eubanks, V. (2018). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press.
6. MIT Media Lab / Algorithmic Justice League. (2019). “About Gender Shades.” https://gendershades.org
7. Stanford Human-Centered AI Institute. (2021). “Racial Equity and AI.” https://hai.stanford.edu