AI Failure Index
AI Algorithmic Decision failures
Scoring and eligibility models that decide who gets credit, benefits, bail, or a job. Failures deny at scale and hide behind the score.
- Incidents
- 45
- Highest severity
- Catastrophic
- Sources cited
- 117
- Newest indexed
- Jun 16, 2026
Social Health Authority AI premiums overcharge poorest Kenyans
Kenya's Social Health Authority deployed an AI-driven predictive model to set health insurance premiums based on income. An investigation found the system systematically overcharged the poorest citizens, effectively denying them access to healthcare.
- Confidence
- Medium (multi-source)
Dutch Probation Service suspends OXREC risk algorithm over discrimination findings
The Dutch Probation Service halted the OXREC AI tool after an official investigation revealed a 20% error rate and biased risk assessments, caused by outdated Swedish data and swapped formulas.
- Confidence
- Medium (multi-source)
ICE AI resume screening error routes recruits to inadequate training
An AI resume-screening tool used by ICE misclassified inexperienced recruits as experienced law enforcement officers. This resulted in approximately 200 hires receiving inadequate online training instead of the required in-person academy course.
- Confidence
- Medium (multi-source)
Instacart AI pricing tests showed shoppers different prices for identical grocery items
A December 2025 study by Consumer Reports, Groundwork Collaborative and More Perfect Union found that Instacart ran AI-driven pricing experiments that resulted in different shoppers seeing different prices for the same items, with some differences reported up to 23%. After public reporting and regulatory questions, Instacart said it would end item price tests on its platform on December 22, 2025. The company had acquired Eversight, an AI pricing and promotions platform, in 2022 and said retailers control prices listed on the app.
- Confidence
- Medium (multi-source)
Thailand freezes 3 million bank accounts in automated anti scam crackdown
The Bank of Thailand froze approximately 3 million bank accounts to combat fraud and mule accounts. The sweeping action resulted in widespread false positives, locking innocent users out of their funds.
- Confidence
- Medium (multi-source)
Brazil AI welfare app wrongly rejects benefit claims
The Brazilian National Social Security Institute's AI-powered app, Meu INSS, wrongly denied benefit claims for hundreds of applicants. The system struggled with complex cases and rural users with low digital literacy, leading to a loss of essential income.
- Confidence
- High (multi-source, primary)
DWP AI fraud detection system found to be biased against vulnerable groups
An AI system used by the UK's Department for Work and Pensions to detect fraud in Universal Credit advance claims was found to be biased. An internal fairness analysis revealed that the system disproportionately flagged certain demographic groups for investigation.
- Confidence
- Medium (multi-source)
Sweden fraud-prediction algorithm found to discriminate against women
Investigative reporting and an Amnesty International statement published on 2024-11-27 found that a fraud risk‑scoring algorithm used by Sweden's Social Insurance Agency produced disproportionate harms to women and other groups. Amnesty called the system discriminatory and urged authorities to discontinue its use. The reporting describes unequal precision and group disparities in the model's risk scores.
- Confidence
- Medium (multi-source)
Home Office AI enforcement tool criticised as rubberstamping immigration decisions
A UK Home Office system called Identify and Prioritise Immigration Cases (IPIC) was criticised by rights groups and privacy researchers in November 2024 as opaque and likely to produce 'rubberstamped' enforcement outcomes. Privacy International obtained redacted manuals and assessments via freedom of information requests that, critics say, show the tool combines sensitive personal data to prioritise cases. Critics warned the system risks bias and poor human oversight in immigration enforcement.
- Confidence
- High (multi-source, primary)
Ticketmaster alleged dynamic pricing caused large Oasis ticket price jumps in 2024
Public complaints after the Oasis ticket sale in September 2024 led the CMA to open an investigation into Ticketmaster’s use of dynamic and tiered pricing and the transparency of price information provided during online queues. The DOJ’s May 2024 antitrust complaint against Live Nation and Ticketmaster raised broader competition concerns. The CMA later secured undertakings from Ticketmaster to improve disclosures while noting it had not found evidence that algorithmic dynamic pricing was used in that specific sale.
- Confidence
- High (multi-source, primary)
Amazon France fined 32 million euros for intrusive employee monitoring
The French regulator CNIL fined Amazon France Logistique €32 million for excessive monitoring of warehouse employees. The system tracked worker interruptions too precisely, violating GDPR data minimization principles.
- Confidence
- High (multi-source, primary)
Dutch tax agency fraud algorithm discriminated against dual nationals
A systemic failure in the Dutch tax authority's fraud-detection algorithms led to discriminatory targeting of dual nationals, causing thousands of families to be wrongly accused and face financial hardship; the event attracted regulatory scrutiny and political repercussions in 2024. The AP AI & Algorithmic Risks Report formally acknowledges systemic AI risks linked to this case.
- Confidence
- High (multi-source, primary)
Plainfield Police Department predictive policing software fails to predict crimes
The Markup and Wired reported that Geolitica's predictive policing software for Plainfield PD produced thousands of predictions with a success rate under 1 percent across 23,631 predictions, and the department stopped using it.
- Confidence
- Medium (multi-source)
RealPage sued by DOJ for using algorithmic pricing to coordinate rent increases
The U.S. Department of Justice filed a civil antitrust lawsuit against RealPage for allegedly using its algorithmic pricing software to facilitate rent collusion among landlords. The government claimed the software allowed landlords to coordinate price increases by sharing competitively sensitive data.
- Confidence
- High (multi-source, primary)
DoorDash faces FTC scrutiny over algorithmic fees and pricing transparency
The Federal Trade Commission investigated DoorDash regarding the use of deceptive and unfair fees in its delivery services. The inquiry focused on pricing transparency and the impact of algorithmic fees on consumers.
- Confidence
- High (multi-source, primary)
Italian Ministry of Education GPS algorithm mis-ranks thousands of teachers
The Italian Ministry of Education's GPS automated allocation system for short-term teachers suffered a critical logic failure. Thousands of eligible teachers were wrongly excluded from assignments, resulting in lost income and numerous lawsuits.
- Confidence
- Medium (multi-source)
Rotterdam welfare fraud model used discriminatory data and performed poorly
A Rotterdam welfare fraud model allegedly used discriminatory data and performed no better than random; two independent outlets describe bias and limited usefulness of the system.
- Confidence
- Medium (multi-source)
Los Angeles scoring system ranks Black and Latino unhoused people lower for subsidized housing
Investigations by The Markup and the Los Angeles Times reported that a scoring system used to prioritize unhoused people for subsidized permanent housing in Los Angeles produced consistently lower priority scores for Black and Latino people. The reporting analysed intake assessment records and found these disparities persisted year after year, making Black and Latino people less likely to receive permanent housing. Subsequent reporting says the city and local agencies moved to change how vulnerability is scored.
- Confidence
- Medium (multi-source)
Palantir Gotham software in Hesse ruled unconstitutional
The German Federal Constitutional Court ruled in February 2023 that Palantir's Gotham software used by the Hesse State Police violated privacy rights. The court suspended mass data analysis due to insufficient legal safeguards.
- Confidence
- High (multi-source, primary)
Southwest Airlines crew-scheduling solver failures cripple holiday flight network
Between December 26-28, 2022 Southwest experienced a large operational collapse where severe weather and failures in crew-scheduling and recovery processes produced widespread cancellations and passenger disruptions. News investigations described the airline’s crew-scheduling solver as unable to restore the network at scale, forcing manual interventions. The U.S. Department of Transportation later assessed penalties and mandated large passenger reimbursements tied to the incident.
- Confidence
- High (multi-source, primary)
VioGén risk-assessment used by Spanish National Police misclassified victims
An academic review and investigative reporting documented transparency, accuracy, and governance problems with VioGén, the Spanish police risk-assessment tool overseen by the Interior Ministry. Reporting and analyses found that the system classified many cases as negligible or low risk and that some victims later suffered repeat attacks or were killed, prompting rights and oversight concerns.
- Confidence
- High (multi-source, primary)
IRCC automated triage pilot flagged for wrongful processing in academic study
IRCC's TRV eApps Advanced Analytics Pilot used AI to triage visa applications. An academic assessment in 2022 found the system lacked accountability and risked wrongful triage.
- Confidence
- Medium (multi-source)
Upstart credit models challenged for disparate impact on minority borrowers
The CFPB revoked a regulatory exemption for Upstart in June 2022 after its AI credit models were challenged for disparate impact on minority borrowers. The controversy centered on the use of educational data in the automated underwriting system.
- Confidence
- High (multi-source, primary)
DWP disability benefits fraud algorithm criticized for lack of transparency
The UK Department for Work and Pensions faced legal challenges over its General Matching Service algorithm used to detect benefit fraud. Critics and disabled people's rights groups alleged the system was unfair and lacked transparency.
- Confidence
- Medium (multi-source)
Haryana Family ID system wrongly declares thousands of living citizens dead
The Haryana government's Parivar Pehchan Patra (PPP) system used AI to automate welfare eligibility, but erroneously marked thousands of living people as deceased. This led to the immediate suspension of critical old-age, widow, and disability pensions for eligible beneficiaries.
- Confidence
- Medium (multi-source)
Gizmodo analysis finds PredPol predictions targeted Black, Latino, and low-income areas
Independent analysis of PredPol prediction logs found the software repeatedly generated predictions concentrated in Black, Latino, and lower-income neighborhoods. The findings, reported by Gizmodo/The Markup and discussed in multiple news outlets, showed patterns consistent with bias arising from the model's training data and operational use.
- Confidence
- High (multi-source, primary)
Foodinho fined 2.6 million euros by Italian regulator over automated rider management
Italy's data protection authority fined Foodinho 2.6 million euros for violating GDPR and labor laws through its automated management of couriers. The regulator found that the company's algorithmic scoring system led to unfair discrimination and lacked human oversight.
- Confidence
- Medium (multi-source)
Twitter Japan suspends accounts of critics of Prime Minister Suga
In June-July 2021 multiple accounts critical of Prime Minister Suga were temporarily frozen by Twitter Japan and later restored. Twitter Japan told reporters the incidents were caused by its AI-powered account-flagging system misidentifying accounts as hijacked or spam. The events drew public criticism and media coverage but no public regulatory enforcement action is documented in the cited sources.
- Confidence
- Medium (multi-source)
Deliveroo Italy algorithm linked to discriminatory rider shift allocations
An Italian court ruled in early January 2021 that an algorithm used by Deliveroo to rate riders and help allocate shifts was discriminatory. Subsequent reporting and Italian prosecutors' actions in February 2026 placed Deliveroo Italy under judicial supervision amid allegations that platform management and algorithmic shift rules contributed to unfair working conditions. Multiple press outlets and an AI incident repository document the ruling and the later supervisory measure.
- Confidence
- High (multi-source, primary)
UK Home Office drops biased visa filtering algorithm
The UK Home Office suspended a visa-streaming tool in August 2020 following allegations of racial bias. The system used nationality to categorize applicants, creating a tiered scrutiny process that disadvantaged specific countries.
- Confidence
- Medium (multi-source)
Gothenburg school placement algorithm uses straight-line distance
The City of Gothenburg's school placement algorithm failed by using straight-line distance instead of actual routing to assign students to schools. This led to incorrect assignments and public outcry in May 2020.
- Confidence
- Medium (multi-source)
Facebook AI content moderation failure causes moderator trauma
Facebook's AI content moderation tools failed to effectively filter harmful content, leading to severe psychological trauma for human moderators. This resulted in a $52 million legal settlement to compensate affected workers.
- Confidence
- High (multi-source, primary)
Dutch government SyRI fraud detection algorithm ruled illegal
The Dutch government used the SyRI algorithm to identify potential social welfare fraud. In February 2020, the District Court of The Hague ruled the system illegal for violating European privacy laws.
- Confidence
- High (multi-source, primary)
Equifax automated credit reporting systems fail to process consumer disputes
Equifax failed to properly investigate consumer credit disputes and inaccurately reported credit scores. The CFPB issued a $15 million penalty for these systemic failures in the company's automated reporting and scoring systems.
- Confidence
- High (multi-source, primary)
Apple Card algorithm allegedly grants lower credit limits to women
Goldman Sachs faced allegations that its Apple Card algorithm discriminated against women. A regulatory probe by the NY DFS followed, though the regulator eventually found no violation of fair lending laws.
- Confidence
- Medium (multi-source)
Study finds Optum risk algorithm understated Black patients' health needs
A 2019 study revealed that Optum's health risk algorithm discriminated against Black patients by substituting health costs for actual health needs. This resulted in a systemic underestimation of risk for Black patients, which limited their access to specialized care management.
- Confidence
- High (multi-source, primary)
US State Education Departments' automated essay scoring found biased against some groups
Automated essay scoring engines were used in many U.S. state standardized tests and multiple investigations and research studies found systematic differences in scores across demographic groups. Reporting and peer-reviewed analysis (including an ETS technical study) showed some engines gave higher average scores to certain groups and lower scores to others, and that some systems could be fooled by nonsense text.
- Confidence
- High (multi-source, primary)
Immigration New Zealand profiles overstayers using predictive data model
In April 2018 reporting revealed Immigration New Zealand had been piloting a data‑modelling programme that used historical demographic and outcome data to build risk profiles of overstayers. Officials described it as a pilot to prioritise cases likely to cause 'harm,' while critics alleged it enabled racial profiling and lacked adequate oversight. The disclosure prompted public debate and scrutiny over the fairness of automated profiling in immigration enforcement.
- Confidence
- High (multi-source, primary)
Parcoursup 2018 rollout drew controversy over opaque and allegedly unfair allocation outcomes
The French national admissions platform Parcoursup was launched in January 2018 to replace the previous centralized system. Within months the rollout generated sustained criticism in major outlets about opacity and allegedly unfair matching outcomes, and subsequent analyses documented how the sequential allocation mechanism and off-platform offers could produce inefficient or surprising assignments. Official reviewers and academic researchers later examined these design features and their consequences.
- Confidence
- High (multi-source, primary)
IBM Watson for Oncology provided unsafe cancer treatment recommendations
IBM Watson for Oncology provided clinically unsafe and incorrect treatment recommendations to healthcare providers. The system allegedly suggested dangerous treatments, such as bleeding drugs for patients with severe hemorrhage.
- Confidence
- Medium (multi-source)
DeepMind and Royal Free NHS Trust process patient records unlawfully
The UK Information Commissioner's Office ruled that DeepMind and the Royal Free NHS Foundation Trust failed to comply with data protection laws. The incident involved the processing of 1.6 million patient records for the Streams app without adequate consent.
- Confidence
- Medium (multi-source)
Researchers find systemic racial bias in PredPol crime forecasting software
A 2016 study revealed that PredPol's predictive policing software produced biased outputs that disproportionately targeted minority communities. The findings indicated that the AI reinforced existing policing patterns rather than predicting actual crime levels.
- Confidence
- High (multi-source, primary)
ProPublica analysis finds COMPAS recidivism risk scores biased against Black defendants
A ProPublica investigation alleged that the COMPAS risk assessment tool exhibited systemic racial bias. The analysis found that Black defendants were flagged as high risk at higher rates than white defendants, even when their actual recidivism rates were similar.
- Confidence
- Medium (multi-source)
Chicago police Heat List criticized for racial bias and ineffectiveness
The Chicago Police Department's Strategic Subject List (SSL), known as the Heat List, was designed to predict individuals likely to be involved in shootings. Independent analysis by Upturn and the RAND Corporation found the system was ineffective at reducing violence and disproportionately targeted individuals based on age and systemic bias.
- Confidence
- Medium (multi-source)
UK Home Office algorithm targets specific nationalities for sham marriage fraud review
The UK Home Office used an automated algorithm to identify potential sham marriages, which was found to be biased against specific nationalities. Legal challenges were brought forward after evidence showed the system disproportionately flagged people from Greece, Albania, Bulgaria, and Romania.
- Confidence
- Medium (multi-source)