DWP fraud detection algorithm targets disabled claimants
The UK DWP used a secretive algorithm to flag Universal Credit claimants for fraud investigations. The system was found to disproportionately target disabled people, which led to a legal challenge and a public admission of bias by the DWP.
The DWP told MPs that bias in the algorithm was necessary for the system to function.
Key facts
- What
- The UK DWP used a secretive algorithm to flag Universal Credit claimants for fraud investigations.
- Incident date
- Dec 1, 2021
- Who
- Department for Work and Pensions (DWP)
- Failure mode
- Brand & Safety Incident
- AI surface
- Algorithmic Decision
- Severity
- High
What happened
The UK Department for Work and Pensions deployed a secretive algorithm to assign risk scores to Universal Credit claimants to identify potential fraud. This system disproportionately flagged disabled people for invasive and grueling investigations. The DWP told MPs that bias in the system was necessary for it to function.
What broke inside the model
- 01 · TriggerA user prompts the model in public view.
- 02 · Model stepThe model produces unsafe or off-brand output.
- 03 · Control gapNo filter holds the line before publish.
- 04 · FailureThe output goes public unchecked.
- 05 · ConsequenceA reputational or safety incident lands.
A contained signal crosses into output that goes public.
The algorithm relied on an opaque risk-scoring mechanism based on claimant data to identify high-probability fraud cases. The admitted bias in the scoring process resulted in the unfair targeting of vulnerable populations.
What it cost
Sources
- PressDWP urged to reveal algorithm that ‘targets’ disabled for benefittheguardian.com
- PressDWP boss admits there is bias in algorithm targeting disabled peoplefoxglove.org.uk
Cite this entry
https://failureindex.ai/failures/dwp-fraud-detection-algorithm-targets-disabledAI Failure Index. "DWP fraud detection algorithm targets disabled claimants" (FI-0672). Realm Labs. https://failureindex.ai/failures/dwp-fraud-detection-algorithm-targets-disabled (indexed Jun 22, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0672. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Realm watches the model's internal state for the signature of unsafe or off-brand generation and can block or reroute the output before it becomes public, in real time rather than after it has been screenshotted.