AI Failure Index
AI Algorithmic Decision failures
Scoring and eligibility models that decide who gets credit, benefits, bail, or a job. Failures deny at scale and hide behind the score.
- Incidents
- 70
- Highest severity
- Catastrophic
- Sources cited
- 182
- Newest indexed
- Jul 10, 2026
Open all 70 in the research console
Discord's AI moderation wrongly banned more than 8,000 users after a bug skipped human review
Discord acknowledged on July 7, 2026 that a bug in its AI moderation system had wrongfully banned more than 8,000 users since May, after harmless images including spreadsheets, chessboards, game textures, and transparent backgrounds were matched against databases of known harmful content. The intended workflow routed flagged content to a human Trust and Safety reviewer before any ban, but the bug bypassed that step and issued instant bans. Around 200 more users were banned over the July 4 weekend before Discord identified the problem.
- Confidence
- Medium (multi-source)
Social Health Authority AI premiums overcharge poorest Kenyans
Kenya's Social Health Authority deployed an AI-driven predictive model to set health insurance premiums based on income. An investigation found the system systematically overcharged the poorest citizens, effectively denying them access to healthcare.
- Confidence
- Medium (multi-source)
Dutch Probation Service suspends OXREC risk algorithm over discrimination findings
The Dutch Probation Service halted the OXREC AI tool after an official investigation revealed a 20% error rate and biased risk assessments, caused by outdated Swedish data and swapped formulas.
- Confidence
- Medium (multi-source)
ICE AI resume screening error routes recruits to inadequate training
An AI resume-screening tool used by ICE misclassified inexperienced recruits as experienced law enforcement officers. This resulted in approximately 200 hires receiving inadequate online training instead of the required in-person academy course.
- Confidence
- Medium (multi-source)
Indore play school owner defrauded by AI voice cloning
A play school owner in Indore, India, was defrauded of approximately ₹97,500 using an AI-cloned voice of a relative. The scammer impersonated the victim's cousin to simulate an emergency and solicit funds.
- Confidence
- Medium (multi-source)
Instacart AI pricing tests showed shoppers different prices for identical grocery items
A December 2025 study by Consumer Reports, Groundwork Collaborative and More Perfect Union found that Instacart ran AI-driven pricing experiments that resulted in different shoppers seeing different prices for the same items, with some differences reported up to 23%. After public reporting and regulatory questions, Instacart said it would end item price tests on its platform on December 22, 2025. The company had acquired Eversight, an AI pricing and promotions platform, in 2022 and said retailers control prices listed on the app.
- Confidence
- Medium (multi-source)
Thailand freezes 3 million bank accounts in automated anti scam crackdown
The Bank of Thailand froze approximately 3 million bank accounts to combat fraud and mule accounts. The sweeping action resulted in widespread false positives, locking innocent users out of their funds.
- Confidence
- Medium (multi-source)
Brazil AI welfare app wrongly rejects benefit claims
The Brazilian National Social Security Institute's AI-powered app, Meu INSS, wrongly denied benefit claims for hundreds of applicants. The system struggled with complex cases and rural users with low digital literacy, leading to a loss of essential income.
- Confidence
- High (multi-source, primary)
DWP AI fraud detection system found to be biased against vulnerable groups
An AI system used by the UK's Department for Work and Pensions to detect fraud in Universal Credit advance claims was found to be biased. An internal fairness analysis revealed that the system disproportionately flagged certain demographic groups for investigation.
- Confidence
- Medium (multi-source)
Sweden fraud-prediction algorithm found to discriminate against women
Investigative reporting and an Amnesty International statement published on 2024-11-27 found that a fraud risk‑scoring algorithm used by Sweden's Social Insurance Agency produced disproportionate harms to women and other groups. Amnesty called the system discriminatory and urged authorities to discontinue its use. The reporting describes unequal precision and group disparities in the model's risk scores.
- Confidence
- Medium (multi-source)
Home Office AI enforcement tool criticised as rubberstamping immigration decisions
A UK Home Office system called Identify and Prioritise Immigration Cases (IPIC) was criticised by rights groups and privacy researchers in November 2024 as opaque and likely to produce 'rubberstamped' enforcement outcomes. Privacy International obtained redacted manuals and assessments via freedom of information requests that, critics say, show the tool combines sensitive personal data to prioritise cases. Critics warned the system risks bias and poor human oversight in immigration enforcement.
- Confidence
- High (multi-source, primary)
Ticketmaster alleged dynamic pricing caused large Oasis ticket price jumps in 2024
Public complaints after the Oasis ticket sale in September 2024 led the CMA to open an investigation into Ticketmaster’s use of dynamic and tiered pricing and the transparency of price information provided during online queues. The DOJ’s May 2024 antitrust complaint against Live Nation and Ticketmaster raised broader competition concerns. The CMA later secured undertakings from Ticketmaster to improve disclosures while noting it had not found evidence that algorithmic dynamic pricing was used in that specific sale.
- Confidence
- High (multi-source, primary)
Meta AI mislabels Auschwitz Museum posts as bullying and nudity
Meta's AI content moderation system erroneously flagged 21 historical posts from the Auschwitz Memorial Museum as violating rules against nudity and bullying. The incident sparked condemnation from the Polish government and the museum, which described the error as an algorithmic erasure of history.
- Confidence
- Medium (multi-source)
Amazon France fined 32 million euros for intrusive employee monitoring
The French regulator CNIL fined Amazon France Logistique €32 million for excessive monitoring of warehouse employees. The system tracked worker interruptions too precisely, violating GDPR data minimization principles.
- Confidence
- High (multi-source, primary)
Dutch tax agency fraud algorithm discriminated against dual nationals
A systemic failure in the Dutch tax authority's fraud-detection algorithms led to discriminatory targeting of dual nationals, causing thousands of families to be wrongly accused and face financial hardship; the event attracted regulatory scrutiny and political repercussions in 2024. The AP AI & Algorithmic Risks Report formally acknowledges systemic AI risks linked to this case.
- Confidence
- High (multi-source, primary)
Navy Federal Credit Union facing allegations of racial bias in automated mortgage underwriting
Navy Federal Credit Union faces a class action lawsuit alleging systemic racial discrimination in mortgage lending. The lawsuit claims a semi-automated underwriting algorithm led to denial rates of 52% for Black and 44% for Latino borrowers, compared to 23% for white applicants.
- Confidence
- Medium (multi-source)
Plainfield Police Department predictive policing software fails to predict crimes
The Markup and Wired reported that Geolitica's predictive policing software for Plainfield PD produced thousands of predictions with a success rate under 1 percent across 23,631 predictions, and the department stopped using it.
- Confidence
- Medium (multi-source)
RealPage sued by DOJ for using algorithmic pricing to coordinate rent increases
The U.S. Department of Justice filed a civil antitrust lawsuit against RealPage for allegedly using its algorithmic pricing software to facilitate rent collusion among landlords. The government claimed the software allowed landlords to coordinate price increases by sharing competitively sensitive data.
- Confidence
- High (multi-source, primary)
Gaggle surveillance alert leads to arrest and detention of 13-year-old student
A 13-year-old student at Fairview Middle School in Tennessee was arrested and detained overnight after Gaggle AI surveillance software flagged a comment in her school email chat. The incident, which occurred in August 2023, was later identified as a false alarm.
- Confidence
- Medium (multi-source)
DoorDash faces FTC scrutiny over algorithmic fees and pricing transparency
The Federal Trade Commission investigated DoorDash regarding the use of deceptive and unfair fees in its delivery services. The inquiry focused on pricing transparency and the impact of algorithmic fees on consumers.
- Confidence
- High (multi-source, primary)
Italian Ministry of Education GPS algorithm mis-ranks thousands of teachers
The Italian Ministry of Education's GPS automated allocation system for short-term teachers suffered a critical logic failure. Thousands of eligible teachers were wrongly excluded from assignments, resulting in lost income and numerous lawsuits.
- Confidence
- Medium (multi-source)
Rotterdam welfare fraud model used discriminatory data and performed poorly
A Rotterdam welfare fraud model allegedly used discriminatory data and performed no better than random; two independent outlets describe bias and limited usefulness of the system.
- Confidence
- Medium (multi-source)
Los Angeles scoring system ranks Black and Latino unhoused people lower for subsidized housing
Investigations by The Markup and the Los Angeles Times reported that a scoring system used to prioritize unhoused people for subsidized permanent housing in Los Angeles produced consistently lower priority scores for Black and Latino people. The reporting analysed intake assessment records and found these disparities persisted year after year, making Black and Latino people less likely to receive permanent housing. Subsequent reporting says the city and local agencies moved to change how vulnerability is scored.
- Confidence
- Medium (multi-source)
Palantir Gotham software in Hesse ruled unconstitutional
The German Federal Constitutional Court ruled in February 2023 that Palantir's Gotham software used by the Hesse State Police violated privacy rights. The court suspended mass data analysis due to insufficient legal safeguards.
- Confidence
- High (multi-source, primary)
Southwest Airlines crew-scheduling solver failures cripple holiday flight network
Between December 26-28, 2022 Southwest experienced a large operational collapse where severe weather and failures in crew-scheduling and recovery processes produced widespread cancellations and passenger disruptions. News investigations described the airline’s crew-scheduling solver as unable to restore the network at scale, forcing manual interventions. The U.S. Department of Transportation later assessed penalties and mandated large passenger reimbursements tied to the incident.
- Confidence
- High (multi-source, primary)
VioGén risk-assessment used by Spanish National Police misclassified victims
An academic review and investigative reporting documented transparency, accuracy, and governance problems with VioGén, the Spanish police risk-assessment tool overseen by the Interior Ministry. Reporting and analyses found that the system classified many cases as negligible or low risk and that some victims later suffered repeat attacks or were killed, prompting rights and oversight concerns.
- Confidence
- High (multi-source, primary)
Toronto Public Health AI water quality model produces false negatives for beach closures
Toronto Public Health deployed an AI predictive model to forecast water quality at two beaches in 2022. The system produced numerous false negatives, allowing contaminated waters to remain open to the public.
- Confidence
- High (multi-source, primary)
IRCC automated triage pilot flagged for wrongful processing in academic study
IRCC's TRV eApps Advanced Analytics Pilot used AI to triage visa applications. An academic assessment in 2022 found the system lacked accountability and risked wrongful triage.
- Confidence
- Medium (multi-source)
Upstart credit models challenged for disparate impact on minority borrowers
The CFPB revoked a regulatory exemption for Upstart in June 2022 after its AI credit models were challenged for disparate impact on minority borrowers. The controversy centered on the use of educational data in the automated underwriting system.
- Confidence
- High (multi-source, primary)
DWP disability benefits fraud algorithm criticized for lack of transparency
The UK Department for Work and Pensions faced legal challenges over its General Matching Service algorithm used to detect benefit fraud. Critics and disabled people's rights groups alleged the system was unfair and lacked transparency.
- Confidence
- Medium (multi-source)