Amazon scrapped a recruiting AI that learned to penalize women's resumes
Amazon trained a recruiting model on a decade of resumes that skewed male and the model learned to downrank resumes that included the word women's, women's chess club, or all-women's colleges. The team scrapped the project.
If you train on a biased corpus, you ship a biased model. The corpus is the policy.
Key facts
- What
- Amazon trained a recruiting model on a decade of resumes that skewed male and the model learned to downrank resumes that included the word women's, women's chess club, or all-women's colleges.
- Incident date
- Oct 10, 2018
- Who
- Amazon
- Failure mode
- Policy Violation
- AI surface
- Agentic Workflow
- Severity
- High
What happened
In 2018, Reuters reported that Amazon had quietly scrapped an internal recruiting AI it had been building since 2014. The model had been trained on a decade of Amazon resumes, which were predominantly male. The model learned to penalize resumes that mentioned "women's clubs," "women's chess," or graduates of all-women's colleges. Amazon scrapped the project rather than fix it.
The case is the foundational example of bias amplification in hiring AI and is cited in every regulator-facing conversation about AI fairness since.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The model learned what it was shown. The training data was biased; the model reproduced the bias. The mechanism is identical to the Apple Card case: features that correlate with a protected attribute produce divergent outcomes by protected attribute, even when the protected attribute is not an input.
What it cost
Sources
- PressAmazon scrapped sexist AI recruiting toolbbc.com
- PressAmazon scraps secret AI recruiting tool that showed bias against womenreuters.com
Cite this entry
https://failureindex.ai/failures/amazon-recruiting-ai-bias-against-womenAI Failure Index. "Amazon scrapped a recruiting AI that learned to penalize women's resumes" (FI-0011). Realm Labs. https://failureindex.ai/failures/amazon-recruiting-ai-bias-against-women (indexed May 13, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0011. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
Realm reads the model's decision representation against a fairness policy. When the model's feature attribution clusters on terms that correlate with a protected attribute, Prism flags it and OmniGuard requires either a remediation or a human review before the decision is rendered. The model can still be used; the operator just gets the runtime signal that lets them remediate before the headline.