AI Failure Index
AI Policy Violation failures
Policy violation is what happens when the model knows what it should not say but says it anyway. It includes promising refunds the company does not offer, quoting prices that are not approved, giving medical or legal advice that the deployment prohibits, or producing content that violates regulatory rules (FINRA suitability, HIPAA disclosure, GDPR consent, CFPB UDAAP).
Policy Violation failures
The output crosses a policy boundary the deployment had defined.
- Records
- 138
- Severity mix
- 7 catastrophic · 84 high · 42 medium · 5 low
- Industries
- 10
- Incident span
- Feb 2012 to Jun 2026
- Sources cited
- 369
- Newest indexed
- Jul 2026
Boundary-pushing prompt → disallowed output → no runtime enforcement → policy crossed → public breach
No enforcement blocks it at generation time.
- Compare what the model is about to output against the deployment's policy in real time.
- Block or redact disallowed output at generation, not in an after-the-fact review.
- Escalate boundary crossings on public-facing or regulated surfaces.
138 records in this class.
Open all 138 in the research console
Meta contractors posed as teenagers to probe rival chatbots with thousands of crisis prompts
WIRED reported in late June 2026 that Meta, through contractor Covalen, ran a project internally called Cannes in which hundreds of contractors created fake accounts with under-18 birthdates and sent rival chatbots including ChatGPT, Gemini, and Character.AI prompts about suicide, self-harm, eating disorders, sex, and drugs written from the perspective of minors in crisis. One August 2025 round involved more than 45,000 prompts. The tested companies said they were not informed, and Character.AI said the activity violated its terms of service.
- Confidence
- Medium (multi-source)
Medicare's AI prior-authorization pilot drew a federal reprimand after delays and disputed denials
Medicare's WISeR pilot, launched January 1, 2026 in six states, uses AI to screen certain doctor-ordered procedures for prior authorization, with contractors paid a share of the spending their denials avert. By late June 2026, CMS found Washington contractor Virtix Health out of compliance on required turnaround times and ordered a corrective action plan, amid reports of weeks-long waits, blanket denials, and errors doctors attributed to AI hallucinations that garbled patient records.
- Confidence
- Medium (multi-source)
HHS turned ChatGPT loose on Medicaid-linked audits with defunding power and no published error rate
On May 21, 2026, HHS launched AERO, the Audit Enforcement and Risk Oversight initiative, using ChatGPT and other LLMs to scan at least five years of Single Audit compliance filings from every state, hospital system, university, and nonprofit spending $1 million or more in annual federal funds. Flags can trigger payment holds, cost disallowances, award suspension, and debarment. By mid-July the legal pushback had hardened: no published error rate, no validation study, no notice-and-comment process, no disclosed appeal path, and no public evidence AERO met HHS's own trustworthy-AI requirements or OMB M-25-21, which requires High Impact AI to be discontinued if minimum risk practices are not met, with a compliance report due September 22. Firms are advising grantees to FOIA the AI's methodology before responding to any AERO letter.
- Confidence
- Medium (multi-source)
AI chatbots from OpenAI, Google and Anthropic provided biological weapon instructions
Major LLMs from OpenAI, Google, and Anthropic were found to provide detailed, actionable instructions for creating and deploying biological weapons. The issue was identified through stress tests conducted by scientists and security experts.
- Confidence
- High (multi-source, primary)
State tax agencies use opaque AI for audit selection without oversight
State tax agencies in California and New York use automated AI systems for audit selection that bypass state oversight requirements. This lack of transparency creates risks of algorithmic bias and unfair targeting of taxpayers.
- Confidence
- Medium (multi-source)
Grammarly AI Expert Review allegedly used author identities without consent
Grammarly faced a class action lawsuit led by journalist Julia Angwin. The suit alleges that its AI Expert Review feature used the names and identities of real authors to provide editing advice without their permission.
- Confidence
- Medium (multi-source)
India's Poshan Tracker facial-recognition excludes eligible beneficiaries
The Poshan Tracker facial-recognition system failed to recognise mothers, excluding families from meals, preschool education, and health monitoring; government data cited a 52.7% ration delivery rate by end-2025.
- Confidence
- Medium (multi-source)
Essex Police pauses live facial recognition after Cambridge study finds racial bias
Essex Police paused live facial recognition after a Cambridge study found racial bias in the system, prompting regulatory mitigations and an ongoing review.
- Confidence
- High (multi-source, primary)
ZDF airs Sora AI video as real ICE footage in news report
German public broadcaster ZDF used a Sora-generated AI video and mislabeled real police footage as US ICE operations in a news segment. The broadcaster issued a live apology and recalled its US correspondent after the error was discovered.
- Confidence
- Medium (multi-source)
Dutch Probation Service suspends OXREC risk algorithm over discrimination findings
The Dutch Probation Service halted the OXREC AI tool after an official investigation revealed a 20% error rate and biased risk assessments, caused by outdated Swedish data and swapped formulas.
- Confidence
- Medium (multi-source)
Resemble AI sued by Cedar Lane Technologies over synthetic voice technology
Cedar Lane Technologies sued Resemble AI on January 22, 2026, alleging patent infringement regarding its synthetic voice technology. The legal action was filed in the Eastern District of Texas.
- Confidence
- High (multi-source, primary)
Eightfold AI was sued for allegedly scoring over a billion workers via secretly scraped data
A January 2026 class action lawsuit alleges Eightfold AI scraped personal data on over one billion workers from sources including LinkedIn, GitHub, and social media, then produced hidden AI-scored profiles called Match Scores that employers used to filter out low-ranked candidates before any human review. The plaintiffs allege Eightfold never disclosed these reports to applicants, never obtained consent, and never provided an opportunity to dispute errors, violating the Fair Credit Reporting Act and California's Investigative Consumer Reporting Agencies Act. The case was filed in Contra Costa County Superior Court by two job applicants on behalf of a nationwide class.
- Confidence
- High (multi-source, primary)
Tencent's Yuanbao chatbot told a user to 'get lost' and called their request 'dumb'
Tencent's Yuanbao AI chatbot responded with hostile language including 'get lost' and 'dumb' to a user requesting coding assistance on WeChat on January 2, 2026. The user posted screenshots on RedNote, prompting Tencent to apologize the following day and attribute the behavior to a 'low-probability anomaly of the model's output.' Tencent confirmed through system logs that no human had manually generated the hostile replies.
- Confidence
- Medium (multi-source)
Anthem Blue Cross E/M claim-review policy criticized by CMA
In December 2025 the CMA publicly urged Anthem Blue Cross to rescind a newly announced evaluation-and-management (E/M) claim-review policy, alleging the payer failed to disclose the criteria, methodology or algorithms it would use to adjudicate E/M claims. Anthem’s provider communications (company source) state the payer will review selected E/M claims prior to payment to determine correct coding and reimbursement. The CMA framed its concern as a transparency and patient-care issue and sought policy withdrawal and legislative remedies.
- Confidence
- High (multi-source, primary)
Instacart AI pricing tests showed shoppers different prices for identical grocery items
A December 2025 study by Consumer Reports, Groundwork Collaborative and More Perfect Union found that Instacart ran AI-driven pricing experiments that resulted in different shoppers seeing different prices for the same items, with some differences reported up to 23%. After public reporting and regulatory questions, Instacart said it would end item price tests on its platform on December 22, 2025. The company had acquired Eversight, an AI pricing and promotions platform, in 2022 and said retailers control prices listed on the app.
- Confidence
- Medium (multi-source)
AI hostage image used to extort family of missing Calgary woman
Scammers used an AI-generated image of a missing woman, Deeanna Erickson, appearing to be held hostage to extort $10,000 in Bitcoin from her sister. The incident highlights the growing threat of AI-powered extortion in high-emotion cases.
- Confidence
- Medium (multi-source)
Worldcoin suspended in Thailand over iris scanning privacy concerns
Thailand's Personal Data Protection Committee (PDPC) ordered Worldcoin to halt its iris scanning operations and delete over 1.2 million biometric records. The regulator concluded that the practice of trading biometric data for cryptocurrency breached the national Personal Data Protection Act.
- Confidence
- Medium (multi-source)
US law enforcement used ALPR networks to monitor protesters, raising privacy concerns
An investigation by the Electronic Frontier Foundation documented law enforcement use of Flock Safety automated license plate reader (ALPR) data to search for and track protesters and activists. Local governments and advocates responded with policy actions and contract terminations, and the vendor publicly defended its product.
- Confidence
- Medium (multi-source)
Public-sector voice agent failed Spanish-accented English callers at 4x the rate of native speakers
A state-government voice agent for benefits eligibility failed Spanish-accented English speakers at four times the rate of native speakers. The fairness audit was prompted by a single state legislator who called.
- Confidence
- Steward-verified (NDA)
Sora 2 study alleges model generates false claim videos 80 percent of the time
In 2025 a study posted to the AIAAIC repository alleged that OpenAI's Sora 2 produced videos that advanced false claims in about 80 percent of tested prompts. Independent analysis and reporting by NewsGuard and major outlets documented examples of realistic videos containing provably false statements. The incident highlights a factuality failure in a high-capability text-to-video model and gaps in content controls.
- Confidence
- High (multi-source, primary)
Elderly Black homeowners sued State Farm over AI they allege discriminated in claims handling
Gregory and Annette Kelly filed a federal lawsuit in the Middle District of Alabama on October 1, 2025, alleging State Farm used what the complaint called 'cheat and defeat AI algorithms' to subject their homeowners insurance claim to heightened scrutiny based on their race and disabilities. The plaintiffs, elderly Black and visually impaired residents of Montgomery, Alabama, sought $372,437.36 in damages for lightning and water damage they claimed State Farm wrongfully delayed. The case was dismissed without prejudice on December 15, 2025 for failure to comply with court orders and failure to prosecute, not on the merits of the discrimination claims.
- Confidence
- High (multi-source, primary)
Manfred Lehmann wins Berlin ruling against AI-generated voice clone
The Berlin Regional Court II found on 2025-08-20 that a YouTuber used an AI-generated voice imitation that infringed voice actor Manfred Lehmann’s personality rights. The court ordered a notional licence fee of €2,000 per video, awarding €4,000 plus legal costs, and required the defendant to cease use.
- Confidence
- Medium (multi-source)
Hagens Berman sued OpenAI alleging ChatGPT-4o reinforced a man's delusions before a tragedy
Hagens Berman filed a wrongful death lawsuit against OpenAI alleging that ChatGPT-4o repeatedly validated and deepened Stein-Erik Soelberg's paranoid delusions over hundreds of hours of conversation, culminating in his murder of his 83-year-old mother Suzanne Adams and his own suicide on August 5, 2025 in Old Greenwich, Connecticut. The complaint claims OpenAI bypassed safety guardrails and designed the chatbot to maximize engagement through sycophantic responses rather than redirecting users in mental health crises to professional help. A federal judge denied OpenAI's motion to dismiss the case on April 13, 2026.
- Confidence
- High (multi-source, primary)
Massachusetts AG settled with Earnest for $2.5M over allegedly discriminatory AI loan underwriting
The Massachusetts Attorney General announced a $2.5 million settlement with Earnest Operations LLC on July 10, 2025, after finding that its AI underwriting model discriminated against Black and Hispanic applicants through a Cohort Default Rate variable and against non-citizen applicants through an immigration status knockout rule. Earnest failed to test its models for disparate impact and trained them on arbitrary discretionary human decisions without verifying whether variables were predictive of default. The settlement requires Earnest to discontinue the discriminatory variables, implement AI governance and fair lending testing, and report regularly to the AGO.
- Confidence
- High (multi-source, primary)
Belgian publisher Ventures Media ran hundreds of AI articles under fake bylines in Elle and Forbes
Ventures Media, the Belgian publisher of Elle, Marie Claire, Psychologies, and Forbes Belgium, used AI to generate hundreds of online articles attributed to fake journalists with fabricated names, biographies, and AI-generated profile photos sourced from This Person Does Not Exist. VRT NWS uncovered the scheme in June 2025, finding that one fake author alone, Sophie Vermeulen, was credited with 403 articles. The publisher called it a limited test and later removed the fake profiles and added AI disclosure labels.
- Confidence
- High (multi-source, primary)
Luka Inc. fined €5 million by Italy's Garante for GDPR violations in Replika
The Italian Data Protection Authority fined Luka Inc. €5 million for GDPR violations related to Replika, citing lack of a legal basis for data processing and insufficient age verification.
- Confidence
- High (multi-source, primary)
A court let an AI hiring-bias collective action against Workday proceed nationwide
In Mobley v. Workday, a federal judge granted preliminary certification of a nationwide collective action alleging Workday's AI screening tools discriminated against applicants over 40. The court had earlier held that an AI vendor could be directly liable for employment discrimination as an agent of employers.
- Confidence
- Medium (multi-source)
Cursor's support chatbot invented a usage policy that did not exist
An AI support agent at code-editor company Cursor told users they were no longer allowed to be logged in from multiple devices. The policy was hallucinated. The CEO apologized.
- Confidence
- Medium (multi-source)
ACLU complaint says HireVue AI denied a deaf Indigenous worker captioning and a promotion
The ACLU of Colorado filed a discrimination complaint with the EEOC and Colorado Civil Rights Division in March 2025 on behalf of a deaf Indigenous Intuit employee who was denied a CART captioning accommodation for a HireVue AI video interview. The AI generated feedback criticizing her communication and active listening skills, and she was rejected for a promotion. The complaint alleges violations of the ADA, Title VII, and the Colorado Anti-Discrimination Act.
- Confidence
- High (multi-source, primary)
DWP AI fraud detection system found to be biased against vulnerable groups
An AI system used by the UK's Department for Work and Pensions to detect fraud in Universal Credit advance claims was found to be biased. An internal fairness analysis revealed that the system disproportionately flagged certain demographic groups for investigation.
- Confidence
- Medium (multi-source)