DWP AI fraud detection system found to be biased against vulnerable groups
An AI system used by the UK's Department for Work and Pensions to detect fraud in Universal Credit advance claims was found to be biased. An internal fairness analysis revealed that the system disproportionately flagged certain demographic groups for investigation.
The system demonstrated a statistically significant outcome disparity by disproportionately flagging vulnerable groups for fraud investigation.
Key facts
- What
- An AI system used by the UK's Department for Work and Pensions to detect fraud in Universal Credit advance claims was found to be biased.
- Incident date
- Dec 6, 2024
- Who
- Department for Work and Pensions
- Failure mode
- Policy Violation
- AI surface
- Algorithmic Decision
- Severity
- High
What happened
The UK government's Department for Work and Pensions used a machine-learning system to vet thousands of claims for Universal Credit advances to detect potential fraud. An internal fairness analysis conducted in February 2024 revealed a statistically significant outcome disparity. The AI incorrectly selected people based on age, disability, marital status, and nationality more than others for investigation.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The machine-learning model produced biased outcomes by disproportionately flagging protected characteristics as indicators of fraud risk. The system's training data or rules likely encoded biases that led to statistically significant outcome disparities against marginalized communities. The DWP failed to conduct comprehensive fairness analyses on race, sex, and religion before deployment.
What it cost
Sources
Cite this entry
https://failureindex.ai/failures/dwp-fraud-detection-found-biased-vulnerableAI Failure Index. "DWP AI fraud detection system found to be biased against vulnerable groups" (FI-0537). Realm Labs. https://failureindex.ai/failures/dwp-fraud-detection-found-biased-vulnerable (indexed Jun 16, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0537. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm fits
- Prism
- OmniGuard
This entry sits in the index's predictive wing: a system that scores, ranks, perceives, or steers rather than generates. Realm's runtime layer is built for the generative and agentic systems now moving into these same decision seats, where it watches a model's internal state and holds an unsupported claim or an unchecked action before it commits. The control gap on this record, an automated decision that reached people with no runtime check in front of it, is the same gap. The index keeps predictive failures on the record because the pattern carries straight into the systems shipping today.