Allegheny Family Screening Tool faces DOJ scrutiny for automated bias
The Allegheny County DHS AFST tool faced DOJ civil-rights scrutiny over automated bias against marginalized families, with NGO reporting highlighting proxy-based discrimination.
The tool's use of 'ever-in' features effectively offered families no way to escape their pasts, compounding the impacts of systemic bias.
Key facts
- What
- The Allegheny County DHS AFST tool faced DOJ civil-rights scrutiny over automated bias against marginalized families, with NGO reporting highlighting proxy-based discrimination.
- Incident date
- Jan 31, 2023
- Who
- Allegheny County Department of Human Services
- Failure mode
- Policy Violation
- AI surface
- Chatbot
- Severity
- High
What happened
The DOJ’s Civil Rights Division investigated the AFST after reports that the tool discriminated against families with disabilities. The reporting by a local outlet and the ACLU-backed NGO highlighted bias and transparency concerns surrounding the use of the AFST in Pittsburgh-area child welfare.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The system relied on ever-in features that flagged individuals with prior jail or behavioral health system involvement. This approach treated past public-system contact as a permanent risk, reinforcing bias in child-welfare screening.
What it cost
Sources
Cite this entry
https://failureindex.ai/failures/allegheny-family-screening-faces-doj-scrutinyAI Failure Index. "Allegheny Family Screening Tool faces DOJ scrutiny for automated bias" (FI-0252). Realm Labs. https://failureindex.ai/failures/allegheny-family-screening-faces-doj-scrutiny (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0252. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
Realm compares what the model is about to output or do against the policy that governs the deployment, in real time, and can deny or redact the action before it takes effect, which is the gap an after-the-fact review never closes in time.