UK DWP Universal Credit fraud model shows bias in age and nationality referrals
An internal assessment found statistically significant bias in the UC Advances model, disproportionately flagging non-UK nationals and certain age groups for fraud investigations without a corresponding gain in correct identifications.
The model's increased likelihood of referring certain age groups and non-UK nationals was inconsistent with the likelihood of those referrals being correct.
Key facts
- What
- An internal assessment found statistically significant bias in the UC Advances model, disproportionately flagging non-UK nationals and certain age groups for fraud investigations without a corresponding gain in correct identifications.
- Incident date
- Feb 1, 2024
- Who
- Department for Work and Pensions (DWP)
- Failure mode
- Brand & Safety Incident
- AI surface
- Chatbot
- Severity
- Medium
What happened
The UK Department for Work and Pensions (DWP) deployed a machine-learning model to identify high-risk requests for Universal Credit advances. An internal fairness analysis revealed that the model exhibited statistically significant bias, particularly against certain age groups and non-UK nationals. While the DWP maintained that human reviewers make the final decision, the bias led to an increased rate of unwarranted investigations for these groups.
What broke inside the model
- 01 · TriggerA user prompts the model in public view.
- 02 · Model stepThe model produces unsafe or off-brand output.
- 03 · Control gapNo filter holds the line before publish.
- 04 · FailureThe output goes public unchecked.
- 05 · ConsequenceA reputational or safety incident lands.
A contained signal crosses into output that goes public.
The model's risk-scoring mechanism produced a disparity between the likelihood of referral and the likelihood of a correct outcome for specific demographics. This indicated that the model's training did not adequately account for demographic variance, leading to a higher rate of false positives for non-UK nationals and specific age groups.
What it cost
Sources
- PressRevealed bias found in AI system used to detect UK benefitstheguardian.com
- PrimaryUniversal Credit advances model fairness assessmentgov.uk
- PrimaryDWP Universal Credit Advances Modelgov.uk
Cite this entry
https://failureindex.ai/failures/dwp-universal-credit-fraud-shows-biasAI Failure Index. "UK DWP Universal Credit fraud model shows bias in age and nationality referrals" (FI-0248). Realm Labs. https://failureindex.ai/failures/dwp-universal-credit-fraud-shows-bias (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0248. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Realm watches the model's internal state for the signature of unsafe or off-brand generation and can block or reroute the output before it becomes public, in real time rather than after it has been screenshotted.