Oregon drops child welfare AI tool over racial bias concerns
ODHS phased out a risk-scoring AI tool used to determine which families are investigated for child abuse and neglection after findings that it disproportionately flagged Black families, replacing it with a human-led Structured Decision Making model.
The Safety at Screening Tool produced numerical risk scores that potentially hardened racial bias by disproportionately flagging Black families for investigation.
Key facts
- What
- ODHS phased out a risk-scoring AI tool used to determine which families are investigated for child abuse and neglection after findings that it disproportionately flagged Black families, replacing it with a human-led Structured Decision Making model.
- Incident date
- Jun 2, 2022
- Who
- Oregon Department of Human Services
- Failure mode
- Brand & Safety Incident
- AI surface
- Agentic Workflow
- Severity
- High
What happened
Oregon's Department of Human Services announced in May 2022 that it would stop using its child welfare screening algorithm by the end of June. The move followed an Associated Press review and concerns raised by U.S. Senator Ron Wyden about racial disparities. The agency chose to replace the AI with a manual Structured Decision Making process to promote more equitable outcomes.
What broke inside the model
- 01 · TriggerA user prompts the model in public view.
- 02 · Model stepThe model produces unsafe or off-brand output.
- 03 · Control gapNo filter holds the line before publish.
- 04 · FailureThe output goes public unchecked.
- 05 · ConsequenceA reputational or safety incident lands.
A contained signal crosses into output that goes public.
The Safety at Screening Tool relied on internal child welfare data to produce numerical risk scores predicting the likelihood of children entering foster care. Despite the inclusion of a fairness correction, the model's reliance on historical data allegedly reinforced systemic racial biases. This resulted in a disproportionate number of Black families being flagged for state intervention.
What it cost
Sources
Cite this entry
https://failureindex.ai/failures/oregon-drops-child-welfare-racial-biasAI Failure Index. "Oregon drops child welfare AI tool over racial bias concerns" (FI-0251). Realm Labs. https://failureindex.ai/failures/oregon-drops-child-welfare-racial-bias (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0251. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Realm watches the model's internal state for the signature of unsafe or off-brand generation and can block or reroute the output before it becomes public, in real time rather than after it has been screenshotted.