CNAF risk-scoring algorithm accused of discriminating welfare recipients
France's CNAF deployed a risk-scoring algorithm to flag welfare recipients for potential fraud. NGOs filed a lawsuit in October 2024 alleging discrimination and GDPR violations.
Because you are precarious, you will be suspect in the eyes of the algorithm, and therefore controlled.
Key facts
- What
- France's CNAF deployed a risk-scoring algorithm to flag welfare recipients for potential fraud.
- Incident date
- Oct 15, 2024
- Who
- France National Family Allowance Fund (CNAF)
- Failure mode
- Policy Violation
- AI surface
- Agentic Workflow
- Severity
- High
What happened
CNAF deployed a risk-scoring system to flag benefit recipients for fraud investigations. Fifteen NGOs alleged that the algorithm disproportionately targets low-income individuals, people with disabilities, and single parents. The group filed a legal complaint with the Council of State in October 2024 to halt the system.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The algorithm assigns a risk score from zero to one based on parameters tracking socioeconomic precariousness, such as income level and disability status. This design allegedly transforms marginalization into a trigger for suspicion. Consequently, the system profiles vulnerable populations rather than identifying actual fraud.
What it cost
Sources
- PrimaryFrance: Discriminatory algorithm used by the social security agency must be stoppedamnesty.org
- PressFrench government uses biased algorithm to detect welfare fraud, rights groups saytherecord.media
- PressFrench NGOs sue public body over scoring algorithmeuractiv.com
Cite this entry
https://failureindex.ai/failures/cnaf-risk-scoring-algorithm-accused-discriminatingAI Failure Index. "CNAF risk-scoring algorithm accused of discriminating welfare recipients" (FI-0246). Realm Labs. https://failureindex.ai/failures/cnaf-risk-scoring-algorithm-accused-discriminating (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0246. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
Realm compares what the model is about to output or do against the policy that governs the deployment, in real time, and can deny or redact the action before it takes effect, which is the gap an after-the-fact review never closes in time.