Sweden fraud-prediction algorithm found to discriminate against women
Investigative reporting and an Amnesty International statement published on 2024-11-27 found that a fraud risk‑scoring algorithm used by Sweden's Social Insurance Agency produced disproportionate harms to women and other groups. Amnesty called the system discriminatory and urged authorities to discontinue its use. The reporting describes unequal precision and group disparities in the model's risk scores.
The model's risk scores were less precise for women and disproportionately flagged them, indicating biased data or features in the fraud‑prediction system.
Key facts
- What
- Investigative reporting and an Amnesty International statement published on 2024-11-27 found that a fraud risk‑scoring algorithm used by Sweden's Social Insurance Agency produced disproportionate harms to women and other groups.
- Incident date
- Nov 27, 2024
- Who
- Försäkringskassan (Swedish Social Insurance Agency)
- Failure mode
- Policy Violation
- AI surface
- Algorithmic Decision
- Severity
- High
What happened
An investigative report by Lighthouse Reports found that a fraud prediction model used by Sweden's social insurance agency assigned risk scores to social security applicants and discriminated against women, migrants and other groups. Amnesty International published a statement on the same day describing the system as discriminatory and urging its immediate discontinuation. Both sources document that the system flagged certain demographic groups at higher rates and showed worse precision for women.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The failure was a fairness and deployment failure: the model and its features produced biased risk scores that were less precise for women, producing disparate impact. Reporting describes opaque model design and feature choices that correlated with protected characteristics, leading to unequal performance across groups. The system's lack of transparency prevented effective mitigation before deployment.
What it cost
Sources
- PressSweden's Suspicion Machinelighthousereports.com
- PressSweden: Authorities must discontinue discriminatory AI systems used by welfare agencyamnesty.org
- PressSweden fraud prediction algorithm found to discriminate against womenaiaaic.org
Cite this entry
https://failureindex.ai/failures/sweden-fraud-prediction-algorithm-found-discriminateAI Failure Index. "Sweden fraud-prediction algorithm found to discriminate against women" (FI-0453). Realm Labs. https://failureindex.ai/failures/sweden-fraud-prediction-algorithm-found-discriminate (indexed Jun 10, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0453. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm fits
- Prism
- OmniGuard
This entry sits in the index's predictive wing: a system that scores, ranks, perceives, or steers rather than generates. Realm's runtime layer is built for the generative and agentic systems now moving into these same decision seats, where it watches a model's internal state and holds an unsupported claim or an unchecked action before it commits. The control gap on this record, an automated decision that reached people with no runtime check in front of it, is the same gap. The index keeps predictive failures on the record because the pattern carries straight into the systems shipping today.