Amazon Scraps Experimental AI Hiring Tool Due to Gender Bias
Amazon developed an AI recruiting tool to score candidates based on resumes but found it systematically discriminated against women. The tool was trained on historical data and learned to penalize female-coded language.
The AI learned to penalize resumes that included the word 'women's' because it was trained on a decade of male-dominated historical data.
Key facts
- What
- Amazon developed an AI recruiting tool to score candidates based on resumes but found it systematically discriminated against women.
- Incident date
- Aug 10, 2016
- Who
- Amazon
- Failure mode
- Brand & Safety Incident
- AI surface
- Algorithmic Decision
- Severity
- Medium
What happened
Amazon developed an AI tool to automate the scoring of job candidates using a five-star rating system. During testing, the company discovered that the algorithm systematically discriminated against women, penalizing resumes that contained words like 'women's'. Amazon subsequently abandoned the experimental project.
What broke inside the model
- 01 · TriggerA user prompts the model in public view.
- 02 · Model stepThe model produces unsafe or off-brand output.
- 03 · Control gapNo filter holds the line before publish.
- 04 · FailureThe output goes public unchecked.
- 05 · ConsequenceA reputational or safety incident lands.
A contained signal crosses into output that goes public.
The AI was trained on resumes submitted to the company over a ten-year period, most of which came from men. Consequently, the model learned to associate success with male-coded language and penalize terms associated with women.
What it cost
Sources
- PressInsight - Amazon scraps secret AI recruiting tool that showed bias against womenreuters.com
- PressAmazon's Gender-Biased Algorithm Is Not Alonebloomberg.com
Cite this entry
https://failureindex.ai/failures/amazon-scraps-experimental-hiring-gender-biasAI Failure Index. "Amazon Scraps Experimental AI Hiring Tool Due to Gender Bias" (FI-0674). Realm Labs. https://failureindex.ai/failures/amazon-scraps-experimental-hiring-gender-bias (indexed Jun 22, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0674. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Realm watches the model's internal state for the signature of unsafe or off-brand generation and can block or reroute the output before it becomes public, in real time rather than after it has been screenshotted.