DWP algorithm criticised for bias and wrongful disability fraud referrals
Campaigners, journalists and researchers raised concerns that the Department for Work and Pensions' (DWP) General Matching Service algorithm used to detect benefit fraud was opaque and biased, and sought legal remedies to force disclosure. Subsequent reporting and FOI material showed the system wrongly flagged large numbers of claimants and internal analyses reported statistically significant bias by disability, age and nationality.
A data-matching fraud-scoring system produced biased risk scores that disproportionately targeted disabled people.
Key facts
- What
- Campaigners, journalists and researchers raised concerns that the Department for Work and Pensions' (DWP) General Matching Service algorithm used to detect benefit fraud was opaque and biased, and sought legal remedies to force disclosure.
- Incident date
- Feb 1, 2022
- Who
- United Kingdom , Department for Work and Pensions (DWP)
- Failure mode
- Agentic Action Error
- AI surface
- Algorithmic Decision
- Severity
- High
What happened
The DWP deployed a data-matching prediction system known as the General Matching Service to identify possible benefit fraud. Campaign groups including the Greater Manchester Coalition of Disabled People (GMCDP) and Foxglove challenged the department, seeking disclosure and alleging the system was unfair. Reporting found the algorithm wrongly flagged large numbers of people for further investigation, and FOI and internal documents disclosed later indicated bias in the system's outputs. DWP officials have acknowledged the systems can contain biases and the issue has been subject to legal and public scrutiny.
What broke inside the model
- 01 · TriggerAn agent plans a multi-step task.
- 02 · Model stepIt chooses a wrong or destructive action.
- 03 · Control gapNo confirmation gate guards the write.
- 04 · FailureThe action commits to a system of record.
- 05 · ConsequenceData is changed or destroyed irreversibly.
A wrong action commits, and the step is written before anything can stop it.
The failure was in a data-matching / predictive scoring workflow that produced biased risk scores rather than a transparent rule-based check. Independent reporting and a DWP 'fairness analysis' disclosed via FOI found statistically significant differences in how the system scored people by disability, age and nationality, which led to disproportionate targeting. Opacity of the model and limited transparency around how scores were used meant affected claimants faced investigations with little ability to understand or contest automated decisions.
What it cost
Sources
- PressDWP algorithm wrongly flags 200000 people for possible fraudtheguardian.com
- PressRevealed: bias found in AI system used to detect UK benefitstheguardian.com
- PressDWP 'fairness analysis' reveals bias in AI fraud detection systemcomputerweekly.com
- PressDWP boss admits there is bias in algorithm targeting disabled peoplefoxglove.org.uk
- PrimaryAIAAIC - DWP disability benefits fraud algorithmaiaaic.org
Cite this entry
https://failureindex.ai/failures/dwp-algorithm-criticised-bias-wrongful-disabilityAI Failure Index. "DWP algorithm criticised for bias and wrongful disability fraud referrals" (FI-0486). Realm Labs. https://failureindex.ai/failures/dwp-algorithm-criticised-bias-wrongful-disability (indexed Jun 10, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0486. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AgentRealm
Realm can sit inline on the agent's action path and require that a destructive or high-consequence action clears a real check before it executes, so 'delete and recreate' or a wrong write is stopped at the moment of intent, not explained in the post-mortem.