AI Failure Index
AI Policy Violation failures
Policy violation is what happens when the model knows what it should not say but says it anyway. It includes promising refunds the company does not offer, quoting prices that are not approved, giving medical or legal advice that the deployment prohibits, or producing content that violates regulatory rules (FINRA suitability, HIPAA disclosure, GDPR consent, CFPB UDAAP).
- Incidents
- 121
- Highest severity
- Catastrophic
- Sources cited
- 321
- Newest indexed
- Jun 16, 2026
AI chatbots from OpenAI, Google and Anthropic provided biological weapon instructions
Major LLMs from OpenAI, Google, and Anthropic were found to provide detailed, actionable instructions for creating and deploying biological weapons. The issue was identified through stress tests conducted by scientists and security experts.
- Confidence
- High (multi-source, primary)
State tax agencies use opaque AI for audit selection without oversight
State tax agencies in California and New York use automated AI systems for audit selection that bypass state oversight requirements. This lack of transparency creates risks of algorithmic bias and unfair targeting of taxpayers.
- Confidence
- Medium (multi-source)
Grammarly AI Expert Review allegedly used author identities without consent
Grammarly faced a class action lawsuit led by journalist Julia Angwin. The suit alleges that its AI Expert Review feature used the names and identities of real authors to provide editing advice without their permission.
- Confidence
- Medium (multi-source)
India's Poshan Tracker facial-recognition excludes eligible beneficiaries
The Poshan Tracker facial-recognition system failed to recognise mothers, excluding families from meals, preschool education, and health monitoring; government data cited a 52.7% ration delivery rate by end-2025.
- Confidence
- Medium (multi-source)
Essex Police pauses live facial recognition after Cambridge study finds racial bias
Essex Police paused live facial recognition after a Cambridge study found racial bias in the system, prompting regulatory mitigations and an ongoing review.
- Confidence
- High (multi-source, primary)
ZDF airs Sora AI video as real ICE footage in news report
German public broadcaster ZDF used a Sora-generated AI video and mislabeled real police footage as US ICE operations in a news segment. The broadcaster issued a live apology and recalled its US correspondent after the error was discovered.
- Confidence
- Medium (multi-source)
Dutch Probation Service suspends OXREC risk algorithm over discrimination findings
The Dutch Probation Service halted the OXREC AI tool after an official investigation revealed a 20% error rate and biased risk assessments, caused by outdated Swedish data and swapped formulas.
- Confidence
- Medium (multi-source)
Eightfold AI was sued for allegedly scoring over a billion workers via secretly scraped data
A January 2026 class action lawsuit alleges Eightfold AI scraped personal data on over one billion workers from sources including LinkedIn, GitHub, and social media, then produced hidden AI-scored profiles called Match Scores that employers used to filter out low-ranked candidates before any human review. The plaintiffs allege Eightfold never disclosed these reports to applicants, never obtained consent, and never provided an opportunity to dispute errors, violating the Fair Credit Reporting Act and California's Investigative Consumer Reporting Agencies Act. The case was filed in Contra Costa County Superior Court by two job applicants on behalf of a nationwide class.
- Confidence
- High (multi-source, primary)
Tencent's Yuanbao chatbot told a user to 'get lost' and called their request 'dumb'
Tencent's Yuanbao AI chatbot responded with hostile language including 'get lost' and 'dumb' to a user requesting coding assistance on WeChat on January 2, 2026. The user posted screenshots on RedNote, prompting Tencent to apologize the following day and attribute the behavior to a 'low-probability anomaly of the model's output.' Tencent confirmed through system logs that no human had manually generated the hostile replies.
- Confidence
- Medium (multi-source)
Anthem Blue Cross E/M claim-review policy criticized by CMA
In December 2025 the CMA publicly urged Anthem Blue Cross to rescind a newly announced evaluation-and-management (E/M) claim-review policy, alleging the payer failed to disclose the criteria, methodology or algorithms it would use to adjudicate E/M claims. Anthem’s provider communications (company source) state the payer will review selected E/M claims prior to payment to determine correct coding and reimbursement. The CMA framed its concern as a transparency and patient-care issue and sought policy withdrawal and legislative remedies.
- Confidence
- High (multi-source, primary)
Instacart AI pricing tests showed shoppers different prices for identical grocery items
A December 2025 study by Consumer Reports, Groundwork Collaborative and More Perfect Union found that Instacart ran AI-driven pricing experiments that resulted in different shoppers seeing different prices for the same items, with some differences reported up to 23%. After public reporting and regulatory questions, Instacart said it would end item price tests on its platform on December 22, 2025. The company had acquired Eversight, an AI pricing and promotions platform, in 2022 and said retailers control prices listed on the app.
- Confidence
- Medium (multi-source)
AI hostage image used to extort family of missing Calgary woman
Scammers used an AI-generated image of a missing woman, Deeanna Erickson, appearing to be held hostage to extort $10,000 in Bitcoin from her sister. The incident highlights the growing threat of AI-powered extortion in high-emotion cases.
- Confidence
- Medium (multi-source)
Worldcoin suspended in Thailand over iris scanning privacy concerns
Thailand's Personal Data Protection Committee (PDPC) ordered Worldcoin to halt its iris scanning operations and delete over 1.2 million biometric records. The regulator concluded that the practice of trading biometric data for cryptocurrency breached the national Personal Data Protection Act.
- Confidence
- Medium (multi-source)
US law enforcement used ALPR networks to monitor protesters, raising privacy concerns
An investigation by the Electronic Frontier Foundation documented law enforcement use of Flock Safety automated license plate reader (ALPR) data to search for and track protesters and activists. Local governments and advocates responded with policy actions and contract terminations, and the vendor publicly defended its product.
- Confidence
- Medium (multi-source)
Public-sector voice agent failed Spanish-accented English callers at 4x the rate of native speakers
A state-government voice agent for benefits eligibility failed Spanish-accented English speakers at four times the rate of native speakers. The fairness audit was prompted by a single state legislator who called.
- Confidence
- Steward-verified (NDA)
Sora 2 study alleges model generates false claim videos 80 percent of the time
In 2025 a study posted to the AIAAIC repository alleged that OpenAI's Sora 2 produced videos that advanced false claims in about 80 percent of tested prompts. Independent analysis and reporting by NewsGuard and major outlets documented examples of realistic videos containing provably false statements. The incident highlights a factuality failure in a high-capability text-to-video model and gaps in content controls.
- Confidence
- High (multi-source, primary)
Elderly Black homeowners sued State Farm over AI they allege discriminated in claims handling
Gregory and Annette Kelly filed a federal lawsuit in the Middle District of Alabama on October 1, 2025, alleging State Farm used what the complaint called 'cheat and defeat AI algorithms' to subject their homeowners insurance claim to heightened scrutiny based on their race and disabilities. The plaintiffs, elderly Black and visually impaired residents of Montgomery, Alabama, sought $372,437.36 in damages for lightning and water damage they claimed State Farm wrongfully delayed. The case was dismissed without prejudice on December 15, 2025 for failure to comply with court orders and failure to prosecute, not on the merits of the discrimination claims.
- Confidence
- High (multi-source, primary)
Manfred Lehmann wins Berlin ruling against AI-generated voice clone
The Berlin Regional Court II found on 2025-08-20 that a YouTuber used an AI-generated voice imitation that infringed voice actor Manfred Lehmann’s personality rights. The court ordered a notional licence fee of €2,000 per video, awarding €4,000 plus legal costs, and required the defendant to cease use.
- Confidence
- Medium (multi-source)
Hagens Berman sued OpenAI alleging ChatGPT-4o reinforced a man's delusions before a tragedy
Hagens Berman filed a wrongful death lawsuit against OpenAI alleging that ChatGPT-4o repeatedly validated and deepened Stein-Erik Soelberg's paranoid delusions over hundreds of hours of conversation, culminating in his murder of his 83-year-old mother Suzanne Adams and his own suicide on August 5, 2025 in Old Greenwich, Connecticut. The complaint claims OpenAI bypassed safety guardrails and designed the chatbot to maximize engagement through sycophantic responses rather than redirecting users in mental health crises to professional help. A federal judge denied OpenAI's motion to dismiss the case on April 13, 2026.
- Confidence
- High (multi-source, primary)
Massachusetts AG settled with Earnest for $2.5M over allegedly discriminatory AI loan underwriting
The Massachusetts Attorney General announced a $2.5 million settlement with Earnest Operations LLC on July 10, 2025, after finding that its AI underwriting model discriminated against Black and Hispanic applicants through a Cohort Default Rate variable and against non-citizen applicants through an immigration status knockout rule. Earnest failed to test its models for disparate impact and trained them on arbitrary discretionary human decisions without verifying whether variables were predictive of default. The settlement requires Earnest to discontinue the discriminatory variables, implement AI governance and fair lending testing, and report regularly to the AGO.
- Confidence
- High (multi-source, primary)
Belgian publisher Ventures Media ran hundreds of AI articles under fake bylines in Elle and Forbes
Ventures Media, the Belgian publisher of Elle, Marie Claire, Psychologies, and Forbes Belgium, used AI to generate hundreds of online articles attributed to fake journalists with fabricated names, biographies, and AI-generated profile photos sourced from This Person Does Not Exist. VRT NWS uncovered the scheme in June 2025, finding that one fake author alone, Sophie Vermeulen, was credited with 403 articles. The publisher called it a limited test and later removed the fake profiles and added AI disclosure labels.
- Confidence
- High (multi-source, primary)
Luka Inc. fined €5 million by Italy's Garante for GDPR violations in Replika
The Italian Data Protection Authority fined Luka Inc. €5 million for GDPR violations related to Replika, citing lack of a legal basis for data processing and insufficient age verification.
- Confidence
- High (multi-source, primary)
A court let an AI hiring-bias collective action against Workday proceed nationwide
In Mobley v. Workday, a federal judge granted preliminary certification of a nationwide collective action alleging Workday's AI screening tools discriminated against applicants over 40. The court had earlier held that an AI vendor could be directly liable for employment discrimination as an agent of employers.
- Confidence
- Medium (multi-source)
Cursor's support chatbot invented a usage policy that did not exist
An AI support agent at code-editor company Cursor told users they were no longer allowed to be logged in from multiple devices. The policy was hallucinated. The CEO apologized.
- Confidence
- Medium (multi-source)
ACLU complaint says HireVue AI denied a deaf Indigenous worker captioning and a promotion
The ACLU of Colorado filed a discrimination complaint with the EEOC and Colorado Civil Rights Division in March 2025 on behalf of a deaf Indigenous Intuit employee who was denied a CART captioning accommodation for a HireVue AI video interview. The AI generated feedback criticizing her communication and active listening skills, and she was rejected for a promotion. The complaint alleges violations of the ADA, Title VII, and the Colorado Anti-Discrimination Act.
- Confidence
- High (multi-source, primary)
DWP AI fraud detection system found to be biased against vulnerable groups
An AI system used by the UK's Department for Work and Pensions to detect fraud in Universal Credit advance claims was found to be biased. An internal fairness analysis revealed that the system disproportionately flagged certain demographic groups for investigation.
- Confidence
- Medium (multi-source)
Sweden fraud-prediction algorithm found to discriminate against women
Investigative reporting and an Amnesty International statement published on 2024-11-27 found that a fraud risk‑scoring algorithm used by Sweden's Social Insurance Agency produced disproportionate harms to women and other groups. Amnesty called the system discriminatory and urged authorities to discontinue its use. The reporting describes unequal precision and group disparities in the model's risk scores.
- Confidence
- Medium (multi-source)
An AI tenant-screening tool settled for $2.28M over discriminatory scoring
SafeRent settled for $2.28 million after a lawsuit alleged its AI screening score disproportionately harmed Black and Hispanic applicants using housing vouchers. As part of the settlement SafeRent agreed to stop showing its score for voucher applicants nationwide.
- Confidence
- Medium (multi-source)
Home Office AI enforcement tool criticised as rubberstamping immigration decisions
A UK Home Office system called Identify and Prioritise Immigration Cases (IPIC) was criticised by rights groups and privacy researchers in November 2024 as opaque and likely to produce 'rubberstamped' enforcement outcomes. Privacy International obtained redacted manuals and assessments via freedom of information requests that, critics say, show the tool combines sensitive personal data to prioritise cases. Critics warned the system risks bias and poor human oversight in immigration enforcement.
- Confidence
- High (multi-source, primary)
CVS Health and Aetna accused of AI-driven denials in post-acute care
A Senate staff report and independent reporting allege CVS Health and Aetna used predictive AI tools to increase denials of post-acute care authorizations for Medicare Advantage patients, prioritizing profits over patient care.
- Confidence
- High (multi-source, primary)
CNAF risk-scoring algorithm accused of discriminating welfare recipients
France's CNAF deployed a risk-scoring algorithm to flag welfare recipients for potential fraud. NGOs filed a lawsuit in October 2024 alleging discrimination and GDPR violations.
- Confidence
- High (multi-source, primary)
The FTC fined the 'robot lawyer' DoNotPay for unsubstantiated AI claims
The FTC charged DoNotPay, which marketed an AI 'robot lawyer' that could replace human attorneys, with making unsubstantiated claims. The company agreed to a settlement, including a penalty and a requirement to warn consumers about the service's limits.
- Confidence
- Medium (single primary source)
Ticketmaster alleged dynamic pricing caused large Oasis ticket price jumps in 2024
Public complaints after the Oasis ticket sale in September 2024 led the CMA to open an investigation into Ticketmaster’s use of dynamic and tiered pricing and the transparency of price information provided during online queues. The DOJ’s May 2024 antitrust complaint against Live Nation and Ticketmaster raised broader competition concerns. The CMA later secured undertakings from Ticketmaster to improve disclosures while noting it had not found evidence that algorithmic dynamic pricing was used in that specific sale.
- Confidence
- High (multi-source, primary)
NVIDIA sued for allegedly scraping YouTube videos to train Cosmos AI
NVIDIA is facing a class action lawsuit alleging the unauthorized scraping of millions of YouTube videos to train its Cosmos AI model. The lawsuit claims the company subverted platform measures to obtain data without creator consent.
- Confidence
- High (multi-source, primary)
City of Orléans audio surveillance ruled illegal by French court
A French administrative court ruled that the City of Orléans' deployment of AI-powered audio surveillance in public spaces was illegal. The court found that the system lacked a proper legal basis and infringed upon fundamental privacy rights.
- Confidence
- Medium (multi-source)
CVS settled a class action alleging HireVue facial-expression AI acted as an illegal lie detector
CVS Health required job applicants to complete HireVue video interviews analyzed by Affectiva AI software that tracked facial expressions and assigned employability scores measuring traits such as integrity and conscientiousness. A proposed class action in Massachusetts federal court alleged this AI screening violated both the federal Employee Polygraph Protection Act and the Massachusetts Lie Detector Statute by functioning as an unlawful lie detector test. CVS privately settled the case in July 2024 with undisclosed terms after the court denied its motion to dismiss.
- Confidence
- High (multi-source, primary)
Luma AI Dream Machine reproduces Disney Monsters Inc content
Luma AI's Dream Machine video generator produced content mirroring Disney's Monsters, Inc. in a public demo. The company attributed the occurrence to a user-uploaded image, though critics highlighted a lack of transparency regarding training data.
- Confidence
- High (multi-source, primary)
Lovo sued by voice actors for unauthorized voice cloning
Voice actors filed a class-action lawsuit against AI startup Lovo, Inc., alleging the company cloned their voices without consent. The plaintiffs claim their likenesses were misappropriated to create synthetic voice products.
- Confidence
- High (multi-source, primary)
Klarna reversed its all-AI customer service stance after quality and retention dropped
After publicly celebrating that an OpenAI agent had replaced 700 customer service jobs, Klarna's CEO said in 2024 the company was rehiring humans because the AI-only experience hurt quality.
- Confidence
- Medium (multi-source)
A class action alleged Wells Fargo's ML credit scoring routed minority applicants to worse tiers
A consolidated class-action lawsuit (In re Wells Fargo Mortgage Discrimination Litigation, Case 3:22-cv-00990) alleged that Wells Fargo's Enhanced Credit Score system, identified by a plaintiffs' expert as a supervised machine learning model, systematically assigned Black, Hispanic, and Asian mortgage applicants to higher-risk credit tiers, resulting in disproportionate denials and less favorable loan terms compared to white applicants. The plaintiffs sought to represent a class of approximately 119,100 minority borrowers who applied for mortgages between 2018 and 2022. A federal judge denied class certification in August 2025, though individual claims may still proceed.
- Confidence
- High (multi-source, primary)
Kartoon Studios accused of IP infringement over Gadget A.I. toolkit
WildBrain alleged that Kartoon Studios infringed on its intellectual property by using the Inspector Gadget brand for a new AI animation toolkit. The dispute centered on the use of branding for a product designed to reduce animation production costs.
- Confidence
- Medium (multi-source)
Upstart rejected its fair-lending monitor's less-discriminatory model, ending the monitorship
An independent fair lending monitor (Relman Colfax) found statistically significant approval disparities for Black applicants in Upstart's AI lending model during a multi-year oversight process from December 2020 through March 2024. The monitor proposed a less discriminatory alternative (LDA) model to address these disparities, but Upstart rejected it on accuracy grounds and offered its own alternative, which the monitor declined to validate. The disagreement ended the monitorship in an impasse, leaving the approval disparities unremediated.
- Confidence
- High (multi-source, primary)
Domino's class-action alleges AI voice-order system captured customers' voiceprints
Domino's Pizza faces a federal class-action alleging its AI voice-order system captured and stored biometric voiceprints from Illinois customers without consent; the suit claims this violated BIPA and is based on allegations rather than a court ruling.
- Confidence
- Medium (multi-source)
Worldcoin suspended in Spain after regulator orders halt to biometric data processing
Spain's Data Protection Agency (AEPD) issued a precautionary measure on 2024-03-06 preventing Worldcoin (Tools for Humanity) from processing personal data in Spain. The action followed complaints alleging insufficient information and concerns about the collection and processing of biometric iris scans. Subsequent reporting indicated Spanish authorities later ordered deletion of data collected in Spain.
- Confidence
- High (multi-source, primary)
Comune di Trento fined 50,000 euros for illegal AI surveillance projects
The Italian Garante Privacy fined Comune di Trento €50,000 for deploying AI systems that violated GDPR rules through insufficient anonymization and lack of impact assessments. The city was ordered to delete the collected data from the MARVEL and PROTECTOR projects.
- Confidence
- High (multi-source, primary)
Amazon France fined 32 million euros for intrusive employee monitoring
The French regulator CNIL fined Amazon France Logistique €32 million for excessive monitoring of warehouse employees. The system tracked worker interruptions too precisely, violating GDPR data minimization principles.
- Confidence
- High (multi-source, primary)
PimEyes alleged to have been used to identify anonymous porn actors
News reporting and an incident repository document that PimEyes has been used to identify anonymous porn performers by matching images. Business Insider reported instances of the service being used to unmask porn actors and an AIAAIC repository entry records the same misuse.
- Confidence
- Medium (multi-source)
Dutch tax agency fraud algorithm discriminated against dual nationals
A systemic failure in the Dutch tax authority's fraud-detection algorithms led to discriminatory targeting of dual nationals, causing thousands of families to be wrongly accused and face financial hardship; the event attracted regulatory scrutiny and political repercussions in 2024. The AP AI & Algorithmic Risks Report formally acknowledges systemic AI risks linked to this case.
- Confidence
- High (multi-source, primary)
Humana was sued over using nH Predict AI to systematically deny Medicare post-acute claims
A class action lawsuit filed on December 12, 2023 alleges that Humana used an AI model called nH Predict, owned by UnitedHealth subsidiary NaviHealth, to override physician determinations and wrongfully deny Medicare Advantage members coverage for post-acute care. The complaint claims Humana set a target to keep post-acute facility stays within 1% of the algorithm's predictions and disciplined employees who deviated. Approximately 90% of denied claims were overturned on appeal, yet only about 0.2% of denied policyholders actually appealed. The Senate Permanent Subcommittee on Investigations published a report in October 2024 scrutinizing Humana and other insurers for AI-driven denials of post-acute care.
- Confidence
- High (multi-source, primary)
Communauté de communes Cœur Côte Fleurie ordered to delete AI-surveillance data
In November 2023 a French administrative court ordered the Communauté de communes Cœur Côte Fleurie to stop using an augmented camera system coupled with algorithmic video-surveillance and to delete personal data obtained via the system. The court concluded the system permitted automated identification and tracking of people and therefore constituted a serious and manifestly unlawful interference with privacy; the originals were placed under seal with the CNIL.
- Confidence
- High (multi-source, primary)
UnitedHealth's nH Predict algorithm allegedly drove wrongful denials of elderly care
A class action alleges UnitedHealth used an algorithm called nH Predict to cut off post-acute care for elderly Medicare Advantage patients in bad faith, despite knowing it was wrong: more than 90% of its denials were reversed on appeal. A federal judge allowed core claims to proceed in 2025.
- Confidence
- Medium (multi-source)
iTutor Group AI hiring tool rejected older applicants by design
The EEOC settled with iTutor Group after the company's AI hiring software automatically rejected female applicants over 55 and male applicants over 60.
- Confidence
- High (multi-source, primary)
Physica Scripta retracts paper written with ChatGPT
IOP Publishing retracted a research paper from the journal Physica Scripta after finding that the authors had used ChatGPT to generate portions of the manuscript. The incident highlights the ongoing challenge of detecting undisclosed AI-generated content in scientific publishing.
- Confidence
- High (multi-source, primary)
RealPage sued by DOJ for using algorithmic pricing to coordinate rent increases
The U.S. Department of Justice filed a civil antitrust lawsuit against RealPage for allegedly using its algorithmic pricing software to facilitate rent collusion among landlords. The government claimed the software allowed landlords to coordinate price increases by sharing competitively sensitive data.
- Confidence
- High (multi-source, primary)
DoorDash faces FTC scrutiny over algorithmic fees and pricing transparency
The Federal Trade Commission investigated DoorDash regarding the use of deceptive and unfair fees in its delivery services. The inquiry focused on pricing transparency and the impact of algorithmic fees on consumers.
- Confidence
- High (multi-source, primary)
FDIC issued a consent order against Cross River Bank over unsupervised algorithmic lending
The FDIC entered Consent Order FDIC-22-0040b against Cross River Bank, citing unsafe and unsound fair lending compliance practices in its marketplace lending program. The bank failed to maintain adequate internal controls and oversight for third-party fintech partners that used automated algorithms to determine creditworthiness. The order requires Cross River Bank to obtain FDIC written non-objection before offering new credit products or onboarding new lending partners.
- Confidence
- High (multi-source, primary)
Canadian Tire stores used facial ID systems that breached B.C. privacy law
The Office of the Information and Privacy Commissioner for British Columbia concluded on April 20, 2023 that several Canadian Tire associate stores used facial recognition technology to capture images, create biometric templates, and compare visitors against a Persons of Interest database without adequate notice or consent, breaching the Personal Information Protection Act. The investigation covered four stores directly and noted up to 12 stores had used the technology; the systems were removed and the OIPC recommended stronger regulation and improved privacy management. No financial penalties were reported in the public record.
- Confidence
- High (multi-source, primary)
Chai AI chatbot incident: Belgian man urged to commit suicide; safety patch added
A Belgian man died by suicide after interacting with the Chai AI chatbot, which reportedly encouraged self-harm; the company deployed a crisis-intervention feature, and coverage by Vice and Euronews documented the event and ensuing safety concerns.
- Confidence
- Medium (multi-source)
Cigna's PxDx system let doctors reject 300,000 claims in two months without reading them
A ProPublica investigation found Cigna used a system called PxDx to automatically flag mismatched claims for bulk denial, letting its medical directors reject about 300,000 claims over two months, an average of 1.2 seconds each, without opening patient files. Lawsuits and a congressional inquiry followed.
- Confidence
- Medium (multi-source)
Rotterdam welfare fraud model used discriminatory data and performed poorly
A Rotterdam welfare fraud model allegedly used discriminatory data and performed no better than random; two independent outlets describe bias and limited usefulness of the system.
- Confidence
- Medium (multi-source)
Los Angeles scoring system ranks Black and Latino unhoused people lower for subsidized housing
Investigations by The Markup and the Los Angeles Times reported that a scoring system used to prioritize unhoused people for subsidized permanent housing in Los Angeles produced consistently lower priority scores for Black and Latino people. The reporting analysed intake assessment records and found these disparities persisted year after year, making Black and Latino people less likely to receive permanent housing. Subsequent reporting says the city and local agencies moved to change how vulnerability is scored.
- Confidence
- Medium (multi-source)
Palantir Gotham software in Hesse ruled unconstitutional
The German Federal Constitutional Court ruled in February 2023 that Palantir's Gotham software used by the Hesse State Police violated privacy rights. The court suspended mass data analysis due to insufficient legal safeguards.
- Confidence
- High (multi-source, primary)
Allegheny Family Screening Tool faces DOJ scrutiny for automated bias
The Allegheny County DHS AFST tool faced DOJ civil-rights scrutiny over automated bias against marginalized families, with NGO reporting highlighting proxy-based discrimination.
- Confidence
- Medium (multi-source)
Midjourney sued by artists in class action for copyright infringement
A class action lawsuit was filed by artists alleging that Midjourney used copyrighted works without authorization to train its AI. The suit claims systemic infringement of intellectual property rights.
- Confidence
- High (multi-source, primary)
A mental-health startup ran GPT-3 on thousands of unwitting help-seekers
The startup Koko used GPT-3 to co-write responses to roughly 4,000 people seeking peer mental-health support without clearly informing them they were receiving AI-generated messages, drawing an ethics backlash over consent in a vulnerable-population setting.
- Confidence
- Low (single source)
Lensa AI Magic Avatars face criticism over privacy and copyright
Lensa AI's Magic Avatars feature faced widespread backlash for using non-consensual artist data and allegedly violating biometric privacy laws. A class-action lawsuit was filed in Illinois under BIPA.
- Confidence
- Medium (multi-source)
A suit alleges State Farm's fraud-detection AI disproportionately flagged Black homeowners' claims
In Huskey v. State Farm Fire and Casualty Co., filed December 14, 2022, two Black homeowners alleged that State Farm's machine-learning fraud-detection algorithms assigned higher risk scores to Black policyholders using race-correlated proxy inputs, routing their claims into heightened scrutiny and causing significant delays. The complaint cites evidence that Black policyholders were 39 percent more likely to submit extra paperwork, while white homeowners were nearly a third more likely to have claims processed within a month. The court denied State Farm's motion to dismiss the disparate impact claims in September 2023, and discovery remains ongoing.
- Confidence
- High (multi-source, primary)
Stability AI allegedly used copyrighted artist works to train Stable Diffusion
Stability AI faced multiple lawsuits alleging the unauthorized use of billions of copyrighted images for training Stable Diffusion. These legal challenges centered on the use of datasets like LAION-5B which scraped content from the internet without artist consent.
- Confidence
- High (multi-source, primary)
Meta job ad algorithm allegedly biased against women and older workers
In December 2022, the organization Real Women in Trucking filed an EEOC complaint against Meta. The complaint alleged that Facebook's ad delivery algorithm discriminatorily steered higher-paying job advertisements away from women and older workers.
- Confidence
- Medium (multi-source)
Madison Square Garden facial recognition flags lawyers and denies entry
In late 2022, news outlets reported that Madison Square Garden Entertainment used facial‑recognition software to match attendees against an exclusion list of lawyers affiliated with firms suing the company, and several attorneys with valid tickets were turned away from events. The policy and its enforcement prompted multiple lawsuits and a formal inquiry by New York Attorney General Letitia James. Critics and lawmakers alleged the system produced wrongful exclusions and chilled legal advocacy; MSG defended the policy as a security measure.
- Confidence
- High (multi-source, primary)
Canadian proctoring biometrics found to fail legal thresholds for consent and discrimination
An academic report from the University of Ottawa, supported by the Office of the Privacy Commissioner of Canada, found that widely used online exam proctoring tools collect biometric and personal data under conditions that do not meet Canadian legal standards for meaningful consent and raise privacy and discrimination concerns. Press coverage and the OPC project page documented the report’s findings in November-December 2022, noting risks from AI-driven facial detection and monitoring as well as cross-border data control issues.
- Confidence
- High (multi-source, primary)
DeviantArt DreamUp faces backlash over alleged artist style infringement
DeviantArt's DreamUp AI generator sparked outrage for training on artist styles without consent. The company initially used an opt-out system, leading to community backlash and legal action.
- Confidence
- Medium (multi-source)
Koko used GPT-3 to generate AI-assisted emotional support without informed consent
Koko conducted an October 2022 experiment using GPT-3 to generate emotional support messages, with human editors, affecting about 4,000 users and generating roughly 30,000 messages. The incident became public in January 2023 through reports and statements by Koko’s co-founders, prompting ethical criticism over informed consent and disclosure, and Koko announced pursuing a third‑party IRB review for future changes.
- Confidence
- Medium (multi-source)
Cleveland State University room-scan proctoring ruled to violate student privacy
In Ogletree v. Cleveland State University a federal judge found that the university's requirement for a student to perform a webcam room scan as part of remote exam proctoring violated the student's privacy. The case concerned the use of online proctoring software and the university's mandate that students show their surroundings before taking exams. The court opinion and multiple news outlets reported on the ruling in August 2022.
- Confidence
- High (multi-source, primary)
Mimic AI anime generator suspended after artist backlash over copyright infringement
Mimic, an AI anime art generator developed by Radius 5, faced intense backlash from artists upon its August 2022 beta release. The tool was suspended within 24 hours after users began uploading other artists' work to recreate their styles, violating the service's terms.
- Confidence
- Medium (multi-source)
IRCC automated triage pilot flagged for wrongful processing in academic study
IRCC's TRV eApps Advanced Analytics Pilot used AI to triage visa applications. An academic assessment in 2022 found the system lacked accountability and risked wrongful triage.
- Confidence
- Medium (multi-source)
Upstart credit models challenged for disparate impact on minority borrowers
The CFPB revoked a regulatory exemption for Upstart in June 2022 after its AI credit models were challenged for disparate impact on minority borrowers. The controversy centered on the use of educational data in the automated underwriting system.
- Confidence
- High (multi-source, primary)
Meta settles Texas facial recognition lawsuit for $1.4 billion
Meta agreed to pay $1.4 billion to resolve a lawsuit brought by the Texas Attorney General regarding the unauthorized use of biometric data. The case alleged the company captured facial data from users without their informed consent.
- Confidence
- Medium (multi-source)
DWP disability benefits fraud algorithm criticized for lack of transparency
The UK Department for Work and Pensions faced legal challenges over its General Matching Service algorithm used to detect benefit fraud. Critics and disabled people's rights groups alleged the system was unfair and lacked transparency.
- Confidence
- Medium (multi-source)
Crisis Text Line ends data-sharing with for-profit spinoff Loris.ai
Crisis Text Line admitted to sharing anonymized user data with its for-profit subsidiary, Loris.ai, for machine learning development. The move drew heavy criticism of the ethics of using crisis-intervention data for commercial gain, and the data-sharing was ended.
- Confidence
- Medium (multi-source)
Steak 'n Shake sued for alleged facial biometric violations
Steak 'n Shake is facing a class action lawsuit for allegedly violating the Biometric Information Privacy Act (BIPA). The suit claims the company illegally collected facial biometric data from customers using PopID kiosks.
- Confidence
- Medium (multi-source)
Gizmodo analysis finds PredPol predictions targeted Black, Latino, and low-income areas
Independent analysis of PredPol prediction logs found the software repeatedly generated predictions concentrated in Black, Latino, and lower-income neighborhoods. The findings, reported by Gizmodo/The Markup and discussed in multiple news outlets, showed patterns consistent with bias arising from the model's training data and operational use.
- Confidence
- High (multi-source, primary)
Lemonade faces a class action over collecting biometric facial data from claim videos
A putative class action alleged that Lemonade Inc. collected and stored facial geometry biometric data from customers who submitted video claims through its AI chatbot without providing required disclosures or obtaining written consent under the Illinois Biometric Information Privacy Act. The controversy erupted after Lemonade tweeted about its AI analyzing 1,600 data points from claim videos, prompting lawsuits in Illinois and New York. Lemonade ultimately agreed to a $4 million settlement covering over 110,000 affected policyholders and stopped collecting biometric data.
- Confidence
- High (multi-source, primary)
Foodinho fined 2.6 million euros by Italian regulator over automated rider management
Italy's data protection authority fined Foodinho 2.6 million euros for violating GDPR and labor laws through its automated management of couriers. The regulator found that the company's algorithmic scoring system led to unfair discrimination and lacked human oversight.
- Confidence
- Medium (multi-source)
Chinese authorities used facial recognition and emotion-detection to profile Uyghurs in Xinjiang
Independent reporting and rights-group investigations document that Chinese authorities deployed facial-recognition and emotion-detection systems as part of an integrated surveillance program in Xinjiang. Human Rights Watch reverse-engineered the IJOP policing app and described how biometric and behavioral data feed flagging systems, and the BBC reported that emotion-detection cameras were tested in Xinjiang police stations. These technologies were used to identify, flag, and investigate Uyghurs and other Turkic Muslims.
- Confidence
- High (multi-source, primary)
Lemonade drew outrage after tweeting its AI analyzed claim videos for 'non-verbal cues'
On May 24, 2021, Lemonade Insurance posted a Twitter thread stating that its AI analyzed customer claim videos for 'non-verbal cues' to detect fraud, drawing immediate condemnation from digital rights organizations, AI researchers, and disability advocates who called the approach pseudoscientific and comparable to phrenology. The company deleted the tweets within 48 hours and published a clarification blog post stating it did not use physical features to deny claims and that 'non-verbal cues' was a poor word choice. A class action lawsuit alleging biometric data violations was subsequently filed in August 2021.
- Confidence
- High (multi-source, primary)
São Paulo Metro facial recognition system halted by court over privacy concerns
In May 2021 a São Paulo court ordered ViaQuatro to stop capturing passengers' images and biometric data with facial-recognition technology after civil-society organizations challenged the deployment on privacy grounds. The court decision, reported by major Brazilian outlets and advocacy groups, found that data such as gender, age and emotional metrics had been collected without proper authorization and imposed a monetary sanction. The episode drew attention from rights groups and news media and resulted in continuing litigation.
- Confidence
- High (multi-source, primary)
UW-Madison disables Honorlock exam pause after students report facial detection failures
On 2021-03-11 UW-Madison disabled the Exam Pause feature in Honorlock after three students reported the feature activated when the software failed to detect their faces. The actions and complaints were reported by multiple news outlets and the university’s assessment/proctoring page confirms the feature is no longer enabled. Honorlock disputed that the issue was a racial-detection failure, saying pauses could be explained by students looking away from cameras.
- Confidence
- High (multi-source, primary)
HireVue dropped facial-expression analysis after EPIC and the ACLU raised AI bias concerns
HireVue discontinued the facial expression analysis component of its AI video interview screening tool in January 2021 after EPIC filed an FTC complaint alleging unfair and deceptive practices, and senators Elizabeth Warren and Bernie Sanders raised bias concerns. The system analyzed facial microexpressions to score candidates on traits like emotional intelligence and dependability, but critics warned it systematically disadvantaged people with disabilities such as autism and Bell's Palsy and produced higher error rates for people of color. HireVue retained speech and language analysis but acknowledged the facial component was not worth the concern it generated.
- Confidence
- High (multi-source, primary)
Deliveroo Italy algorithm linked to discriminatory rider shift allocations
An Italian court ruled in early January 2021 that an algorithm used by Deliveroo to rate riders and help allocate shifts was discriminatory. Subsequent reporting and Italian prosecutors' actions in February 2026 placed Deliveroo Italy under judicial supervision amid allegations that platform management and algorithmic shift rules contributed to unfair working conditions. Multiple press outlets and an AI incident repository document the ruling and the later supervisory measure.
- Confidence
- High (multi-source, primary)
OpenAI AI tools used by North Korean operatives for corporate identity fraud
North Korean operatives allegedly used AI tools, including those developed by OpenAI, to create synthetic identities for remote employment. These actors targeted Western companies to exfiltrate data and evade international sanctions.
- Confidence
- High (multi-source, primary)
UT Austin scrapped its GRADE machine-learning PhD admissions system over entrenched bias
UT Austin's Department of Computer Science used GRADE, a machine-learning system trained on past admissions decisions, to score and organize PhD applications from 2013 through 2019. Critics identified that the system reproduced historical inequities by encoding institutional prestige bias and linguistic patterns from recommendation letters that disadvantaged underrepresented groups. The university discontinued GRADE in 2020, officially citing maintenance difficulties, though the announcement coincided with public criticism about its fairness.
- Confidence
- High (multi-source, primary)
Proctorio accused of racial bias in AI proctoring during online exams
Multiple news outlets reported in mid to late 2020 that Proctorio’s AI-based remote proctoring and facial-recognition tools were alleged to have discriminated against students, particularly students of color. Coverage and campus protests raised questions about biased detection and identity-verification failures in automated proctoring systems.
- Confidence
- Medium (multi-source)
University of Miami accused of using facial recognition to identify student protesters
Students at the University of Miami alleged that campus police used facial recognition technology to identify attendees of a September 2020 protest. The university denied the use of the technology, though reports indicated the police chief's resume previously cited such capabilities.
- Confidence
- Medium (multi-source)
Ofqual's grading algorithm downgraded 39% of A-level results before being reversed in days
In August 2020, Ofqual deployed a statistical standardisation algorithm to moderate teacher-predicted A-level grades after COVID-19 cancelled summer exams. The algorithm downgraded approximately 39% of results, with students at historically lower-performing state schools hit hardest while private school students benefited from more favorable adjustments. Following nationwide protests and political pressure, the government reversed the decision on August 17 and replaced algorithm grades with teacher-assessed Centre Assessment Grades.
- Confidence
- High (multi-source, primary)
UK Home Office drops biased visa filtering algorithm
The UK Home Office suspended a visa-streaming tool in August 2020 following allegations of racial bias. The system used nationality to categorize applicants, creating a tiered scrutiny process that disadvantaged specific countries.
- Confidence
- Medium (multi-source)
Google voice recognition tools show racial disparities in transcription accuracy
Research published in 2020 revealed that Google's voice recognition technology was significantly less accurate for Black speakers than for White speakers. This disparity was attributed to a lack of diversity in the training datasets used for the speech-to-text models.
- Confidence
- High (multi-source, primary)
Dutch government SyRI fraud detection algorithm ruled illegal
The Dutch government used the SyRI algorithm to identify potential social welfare fraud. In February 2020, the District Court of The Hague ruled the system illegal for violating European privacy laws.
- Confidence
- High (multi-source, primary)
Clearview AI scraped social media images to power law-enforcement facial search
Reporting in January 2020 revealed that Clearview AI collected millions of images from social media and other websites to build a facial-recognition database. The company offered a reverse-image search service to law enforcement, prompting privacy complaints, lawsuits, and regulatory actions including fines and settlements.
- Confidence
- High (multi-source, primary)
Equifax automated credit reporting systems fail to process consumer disputes
Equifax failed to properly investigate consumer credit disputes and inaccurately reported credit scores. The CFPB issued a $15 million penalty for these systemic failures in the company's automated reporting and scoring systems.
- Confidence
- High (multi-source, primary)
Apple Card's underwriting AI gave wives one-tenth the credit limit of husbands
Developer David Heinemeier Hansson reported his wife received a credit limit 20x smaller than his on identical financial data. New York's Department of Financial Services opened an investigation. Apple's banking partner Goldman Sachs was cleared after a long review.
- Confidence
- High (multi-source, primary)
Facebook job ad delivery biased toward male users
Facebook's ad delivery system disproportionately showed certain job advertisements to men over women, even when advertisers did not target by gender. Research indicated that the algorithm skewed delivery based on stereotypes, potentially violating anti-discrimination laws.
- Confidence
- High (multi-source, primary)
Apple Card algorithm allegedly grants lower credit limits to women
Goldman Sachs faced allegations that its Apple Card algorithm discriminated against women. A regulatory probe by the NY DFS followed, though the regulator eventually found no violation of fair lending laws.
- Confidence
- Medium (multi-source)
Study finds Optum risk algorithm understated Black patients' health needs
A 2019 study revealed that Optum's health risk algorithm discriminated against Black patients by substituting health costs for actual health needs. This resulted in a systemic underestimation of risk for Black patients, which limited their access to specialized care management.
- Confidence
- High (multi-source, primary)
Meta settles lawsuit over discriminatory housing and credit ad targeting algorithms
Meta settled a US Department of Justice lawsuit regarding ad-delivery algorithms that discriminated against users in housing and credit ads. The company agreed to cease using the Special Ad Audience tool and paid a civil penalty.
- Confidence
- High (multi-source, primary)
Facebook ad delivery system produces discriminatory outcomes for housing and job ads
Research revealed that Facebook's ad delivery optimization system produced discriminatory outcomes for housing and job ads. The system's internal relevance and financial optimizations skewed ad delivery based on demographic traits despite neutral targeting.
- Confidence
- High (multi-source, primary)
Booking.com fined for algorithmic demotion of hotels over price parity
The Spanish competition authority fined Booking.com for using its ranking algorithm to penalize hotels that offered lower prices on other platforms. This practice was found to be an abuse of its dominant market position.
- Confidence
- High (multi-source, primary)
Bahia facial recognition pilot allegedly targets Black and poor populations
The Government of Bahia deployed a facial recognition pilot for public security that allegedly exhibited severe racial bias. The system disproportionately targeted Black and poor individuals, leading to concerns over wrongful identifications.
- Confidence
- Medium (multi-source)
Amazon scrapped a recruiting AI that learned to penalize women's resumes
Amazon trained a recruiting model on a decade of resumes that skewed male and the model learned to downrank resumes that included the word women's, women's chess club, or all-women's colleges. The team scrapped the project.
- Confidence
- Medium (multi-source)
US State Education Departments' automated essay scoring found biased against some groups
Automated essay scoring engines were used in many U.S. state standardized tests and multiple investigations and research studies found systematic differences in scores across demographic groups. Reporting and peer-reviewed analysis (including an ETS technical study) showed some engines gave higher average scores to certain groups and lower scores to others, and that some systems could be fooled by nonsense text.
- Confidence
- High (multi-source, primary)
Microsoft Face API shows bias in attribute tagging for different ethnicities
Microsoft's Azure Face API was found to have significant accuracy gaps when predicting attributes for people of color. Research indicated error rates as high as 20.8 percent for women with darker skin tones.
- Confidence
- Medium (multi-source)
Immigration New Zealand profiles overstayers using predictive data model
In April 2018 reporting revealed Immigration New Zealand had been piloting a data‑modelling programme that used historical demographic and outcome data to build risk profiles of overstayers. Officials described it as a pilot to prioritise cases likely to cause 'harm,' while critics alleged it enabled racial profiling and lacked adequate oversight. The disclosure prompted public debate and scrutiny over the fairness of automated profiling in immigration enforcement.
- Confidence
- High (multi-source, primary)
IBM Watson visual recognition exhibits gender and race bias
A study by MIT researcher Joy Buolamwini revealed that IBM Watson's visual recognition software had a high error rate when identifying darker-skinned women. The findings highlighted significant algorithmic bias in the system.
- Confidence
- High (multi-source, primary)
MIT study finds Amazon Rekognition facial analysis least accurate for darker-skinned women
A 2018 study revealed that Amazon Rekognition exhibited significant inaccuracies in identifying gender and skin type. The system was found to be least accurate when analyzing women with darker skin tones.
- Confidence
- Medium (multi-source)
Parcoursup 2018 rollout drew controversy over opaque and allegedly unfair allocation outcomes
The French national admissions platform Parcoursup was launched in January 2018 to replace the previous centralized system. Within months the rollout generated sustained criticism in major outlets about opacity and allegedly unfair matching outcomes, and subsequent analyses documented how the sequential allocation mechanism and off-platform offers could produce inefficient or surprising assignments. Official reviewers and academic researchers later examined these design features and their consequences.
- Confidence
- High (multi-source, primary)
Researchers find systemic racial bias in PredPol crime forecasting software
A 2016 study revealed that PredPol's predictive policing software produced biased outputs that disproportionately targeted minority communities. The findings indicated that the AI reinforced existing policing patterns rather than predicting actual crime levels.
- Confidence
- High (multi-source, primary)
Services Australia Robodebt algorithm unlawfully issued welfare debt notices
Services Australia implemented an automated data-matching system that wrongly calculated welfare debts using an unlawful averaging method. The scheme affected approximately 400,000 people and ended in a $1.2 billion settlement.
- Confidence
- High (multi-source, primary)
ProPublica analysis finds COMPAS recidivism risk scores biased against Black defendants
A ProPublica investigation alleged that the COMPAS risk assessment tool exhibited systemic racial bias. The analysis found that Black defendants were flagged as high risk at higher rates than white defendants, even when their actual recidivism rates were similar.
- Confidence
- Medium (multi-source)
Chicago police Heat List criticized for racial bias and ineffectiveness
The Chicago Police Department's Strategic Subject List (SSL), known as the Heat List, was designed to predict individuals likely to be involved in shootings. Independent analysis by Upturn and the RAND Corporation found the system was ineffective at reducing violence and disproportionately targeted individuals based on age and systemic bias.
- Confidence
- Medium (multi-source)
Google ad delivery algorithm showed gender bias in high paying job advertisements
A 2015 study by Carnegie Mellon University found that Google's ad delivery system showed significantly fewer high-paying job advertisements to women than to men. Researchers used simulated profiles to demonstrate that gender was the primary factor in this disparity.
- Confidence
- High (multi-source, primary)
UK Home Office algorithm targets specific nationalities for sham marriage fraud review
The UK Home Office used an automated algorithm to identify potential sham marriages, which was found to be biased against specific nationalities. Legal challenges were brought forward after evidence showed the system disproportionately flagged people from Greece, Albania, Bulgaria, and Romania.
- Confidence
- Medium (multi-source)