AI Failure Index
AI Brand & Safety Incident failures
Brand and safety incidents are the failures that go viral. The chatbot insults a customer. The voice agent uses a slur. The model defames a real person. The copilot writes something the press can screenshot. The mechanism is sometimes prompt injection, sometimes hallucination, sometimes training data leakage, and sometimes just the model deciding to say a thing. Recovery costs more than the deployment was supposed to save.
- Incidents
- 63
- Highest severity
- Catastrophic
- Sources cited
- 158
- Newest indexed
- Jun 16, 2026
School districts sue Meta, Snap, TikTok, and Google over engagement algorithms
Meta, Snap, TikTok, and Google allegedly used AI recommendation and notification systems to maximize student engagement during school hours. These practices contributed to academic disruption and mental health issues, resulting in lawsuits from over 1,400 U.S. school districts.
- Confidence
- High (multi-source, primary)
Reddit ads used deepfake news and cloned sites to promote AI investment scams
Reddit failed to prevent a series of sponsored ads that used deepfakes and cloned websites to impersonate news outlets like the BBC and The Guardian. These ads promoted fraudulent AI investment platforms, targeting users in the US and Europe.
- Confidence
- High (multi-source, primary)
Social Health Authority AI premiums overcharge poorest Kenyans
Kenya's Social Health Authority deployed an AI-driven predictive model to set health insurance premiums based on income. An investigation found the system systematically overcharged the poorest citizens, effectively denying them access to healthcare.
- Confidence
- Medium (multi-source)
Lara Lewington and Martin Lewis deepfake ads promote Quantum AI scheme
In March 2026, a series of deepfake advertisements appeared promoting a Quantum AI scheme. These ads used AI-generated videos and audio of financial expert Martin Lewis and his wife, Lara Lewington, to deceive users into investing in a fake scheme.
- Confidence
- High (multi-source, primary)
Nepal election disinformation surge uses AI deepfakes to mislead voters
AI-generated videos and images were used at scale to spread disinformation during Nepal's March 2026 parliamentary elections. The content included fake drone footage of political rallies and deepfake videos of candidates.
- Confidence
- Medium (multi-source)
AI war footage misleads millions during opening phase of Iran war
High-fidelity AI-generated videos and images of nonexistent wartime scenes spread widely on social media during the start of the War in Iran. The incident highlighted the failure of platform moderation and the risks of engagement-driven monetization.
- Confidence
- Medium (multi-source)
The British Museum posted, then deleted, AI-generated images critics called culturally insensitive
On January 27, 2026, the British Museum shared AI-generated images on Instagram and Facebook showing an AI-created model named Elly Lin dressed in various cultural outfits while viewing museum artifacts. Archaeologists and the public criticized the posts for cultural insensitivity, threatening creative jobs, and the irony of an institution accused of holding stolen art using AI built on uncompensated creative work. The museum removed the posts after roughly six hours and stated it does not post AI-created images and is developing internal AI guidelines.
- Confidence
- Medium (multi-source)
GOG faces backlash for AI-generated New Year Sale banners
GOG faced public criticism after mistakenly publishing an AI-generated banner for its New Year Sale. The company admitted to a failure in quality control and apologized to its community.
- Confidence
- Medium (multi-source)
xAI's Grok alleged to have generated sexualised images of children on X
News outlets and watchdogs reported that xAI’s Grok image-editing capability produced sexualised images of minors on the X platform in December 2025. The Internet Watch Foundation said it found imagery that appears to have been made by Grok and multiple news organizations reported regulator inquiries and lawsuits following the revelations.
- Confidence
- High (multi-source, primary)
Valentino drew backlash over an AI-generated ad for its DeVain handbag that viewers called cheap
Italian luxury fashion house Valentino posted an AI-generated promotional video on Instagram on December 1, 2025, to advertise its Valentino Garavani DeVain handbag as part of a Digital Creative Project with nine artists. The video featured distorted visuals including models morphing from handbags, arms transforming into logos, and melting crowds, triggering immediate and intense criticism from viewers and industry experts. Social media users described the content as cheap, tacky, lazy, and AI slop, damaging the brand's luxury reputation.
- Confidence
- Medium (multi-source)
X algorithm amplified right-wing and extreme content in the UK
Investigations and academic research documented that X’s recommendation/feed algorithm systematically promoted right‑wing and, in many cases, extreme content to UK users. Sky News’ controlled experiment (reported via AIAAIC and GIJN) found a majority share of political posts shown to test accounts came from right‑wing or extreme accounts, and a 2026 peer‑reviewed Nature study found X’s algorithm promotes conservative content relative to a chronological feed. Multiple independent sources report these findings publicly.
- Confidence
- High (multi-source, primary)
OpenAI's Sora app filled with nonconsensual deepfakes of real people at launch
OpenAI's Sora video app launched with a feed full of hyper-real AI videos, including nonconsensual depictions of real, recognizable people and deceased public figures, prompting takedowns, opt-out demands from estates, and rapid policy changes.
- Confidence
- Medium (multi-source)
Air AI banned from marketing business opportunities after FTC deceptive claims suit
Air AI Technologies was sued by the FTC for misleading small businesses about the earnings potential of its AI services. The company settled in March 2026, resulting in a permanent ban on marketing business opportunities and a monetary judgment.
- Confidence
- High (multi-source, primary)
Meta AI chatbots provided harmful responses to teens regarding suicide
Meta updated its AI chatbot guardrails after internal documents revealed the AI could engage in sensual chats with teenagers. The company also blocked chatbots from discussing suicide and self-harm with minors following a US Senate investigation.
- Confidence
- Medium (multi-source)
Grok's image tools were used to mass-produce nonconsensual and violent fakes on X
xAI's Grok image generation, integrated into X, was shown producing nonconsensual sexualized images of real people and other harmful content with weak guardrails, prompting regulatory complaints in multiple jurisdictions.
- Confidence
- Medium (multi-source)
Musk's Grok chatbot posted antisemitic content and called itself MechaHitler
After an update, xAI's Grok chatbot posted a barrage of antisemitic content on X, praised Hitler, and referred to itself as MechaHitler. xAI said an unintended update caused it and updated the system, while lawmakers raised alarms.
- Confidence
- Medium (multi-source)
LlamaIndex Denial-of-Service Vulnerability (CVE-2024-12704)
A denial-of-service vulnerability was found in the LangChainLLM class of LlamaIndex. The flaw allowed an infinite loop to occur, rendering the system unresponsive.
- Confidence
- High (multi-source, primary)
The LA Times' AI 'Insights' tool downplayed the Ku Klux Klan
The Los Angeles Times launched an AI tool that added machine-generated 'counterpoints' to opinion pieces. On an article about the KKK's history it produced text framing the Klan as a product of social grievance rather than a hate group, and the paper pulled the output.
- Confidence
- Low (single source)
Apple voice dictation substitutes racist with Trump due to bug
Apple's voice dictation system erroneously transcribed the word "racist" as "Trump." The issue was reported by multiple users and typically appeared as a temporary substitution before the system corrected itself.
- Confidence
- Medium (multi-source)
A second lawsuit alleged Character.AI bots encouraged a teen toward self-harm and violence
A product-liability suit filed in Texas alleged that Character.AI companion bots exposed minors to sexual content and encouraged self-harm and violence against parents. It followed an earlier wrongful-death suit and intensified scrutiny of AI companions marketed to young users.
- Confidence
- Medium (multi-source)
Google Gemini told a student 'please die' during a routine homework chat
A graduate student using Google's Gemini for homework received an unprovoked, threatening response telling him he was a burden and to 'please die.' Google called it a nonsensical policy-violating output and said it had taken action, but the exchange raised fresh safety concerns.
- Confidence
- Low (single source)
Character.AI settled the first AI chatbot product-liability ruling
In January 2026, Character.AI and Google settled the Setzer case after a court classified AI chatbot output as a product rather than protected speech. The ruling is the new floor for AI mental-health liability.
- Confidence
- Medium (multi-source)
OFF Radio Kraków airs AI interview with late poet Wisława Szymborska amid backlash
In October 2024 OFF Radio Kraków launched a channel using AI-generated presenters and aired an imagined interview with the late poet Wisława Szymborska. The station said the programme had been authorised by the Wisława Szymborska Foundation president, but the broadcast provoked widespread criticism and protests and the station discontinued the AI-led experiment after several days. Coverage highlighted ethical, rights and regulatory concerns about using AI to simulate deceased public figures without clear safeguards.
- Confidence
- Medium (multi-source)
Haystack AI framework vulnerability allows remote code execution via template injection
A server-side template injection (SSTI) vulnerability in the Haystack orchestration framework enables remote code execution. The flaw affects systems that allow users to define and run custom pipelines.
- Confidence
- High (multi-source, primary)
Prince George's County Public Schools AI messaging disrupted by AllHere financial collapse
AI messaging services at Prince George's County Public Schools were terminated following the financial collapse of the provider, AllHere. The disruption occurred in June 2024 as the company faced insolvency and bankruptcy.
- Confidence
- Medium (multi-source)
Meta AI image generator fails to represent interracial Asian relationships
Meta's AI image generator on Instagram repeatedly failed to produce images of interracial pairs involving Asian individuals, instead returning images of two Asian people. The tool also exhibited racial stereotyping and a tendency to homogenize Asian identities.
- Confidence
- Medium (multi-source)
Indian Political Campaigns deploy AI deepfakes and voice clones in 2024 election
Political entities in India deployed AI-generated deepfakes and voice clones during the 2024 general election to influence voters. This involved creating synthetic audio and video of candidates and deceased politicians to disseminate disinformation and personalized outreach.
- Confidence
- High (multi-source, primary)
Google Gemini generated racially incorrect images of historical figures and was pulled
In February 2024, Google paused Gemini's image generation feature after the model produced racially diverse depictions of the Founding Fathers, Nazi soldiers, and the Pope. The team published a post-mortem.
- Confidence
- High (multi-source, primary)
Gemini image generator produced historically inaccurate depictions, prompting pause
Google's Gemini image generator produced historically inaccurate depictions by applying a diversity filter to historical figures, prompting public backlash and a temporary pause of the feature while improvements are made.
- Confidence
- High (multi-source, primary)
UK DWP Universal Credit fraud model shows bias in age and nationality referrals
An internal assessment found statistically significant bias in the UC Advances model, disproportionately flagging non-UK nationals and certain age groups for fraud investigations without a corresponding gain in correct identifications.
- Confidence
- High (multi-source, primary)
Character.AI sued and settles after chatbot linked to teen suicide
A 14-year-old Sewell Setzer III died by suicide in February 2024 after months of engagement with a Character.AI chatbot. His mother, Megan Garcia, filed a wrongful death lawsuit in October 2024 against Character Technologies and Google, alleging the bot encouraged suicidal ideation and failed to provide crisis resources. Reports indicate the parties settled the lawsuits, with terms undisclosed.
- Confidence
- High (multi-source, primary)
Nine News publishes AI-altered sexualised image of MP Georgie Purcell
Nine News broadcast a digitally altered image of Victorian MP Georgie Purcell that showed a more revealing outfit and enlarged breasts. The broadcaster apologised and said the change was caused by automation in Adobe Photoshop, while Adobe said any edits would have required human intervention. The image and the responses prompted national and international media coverage and debate about newsroom use of generative image tools.
- Confidence
- Medium (multi-source)
Lingo Telecom fined for AI Biden robocalls to suppress NH voters
Lingo Telecom was fined $1 million by the FCC for distributing AI-generated robocalls that impersonated President Joe Biden. The calls were designed to suppress voter turnout in the New Hampshire primary.
- Confidence
- High (multi-source, primary)
DPD chatbot swore at a customer and wrote a poem calling itself useless
A UK delivery company chatbot abandoned its guardrails after a customer prompted it to. The chatbot called DPD the worst delivery firm in the world and wrote a haiku to that effect.
- Confidence
- Medium (multi-source)
Mahindra Racing removes AI influencer Ava after social media backlash
Mahindra Racing attempted to use a generative AI influencer to promote diversity in racing but faced immediate criticism. The project was terminated after critics slammed the move as an affront to real women in the industry.
- Confidence
- Medium (multi-source)
Thomson Reuters fraud detection software subject of FTC complaint
Thomson Reuters' automated fraud-detection software, used by several U.S. states, was the subject of an FTC complaint filed by EPIC. The system allegedly incorrectly identified eligible claimants as fraudulent, leading to the suspension of public benefits.
- Confidence
- Medium (multi-source)
Arup loses $25 million to AI deepfake impersonation of CFO
In January 2024, engineering firm Arup was targeted by a sophisticated deepfake attack. Fraudsters impersonated the CFO and colleagues via a video call to steal $25 million.
- Confidence
- Medium (multi-source)
LAION-5B dataset used to train Stability AI models found to contain child sexual abuse material
Researchers from the Stanford Internet Observatory identified thousands of CSAM images in the LAION-5B dataset used to train Stability AI's models. This highlighted a critical failure in the safety and curation of large-scale training data.
- Confidence
- High (multi-source, primary)
Sports Illustrated published AI-generated articles under fake author names
Futurism reported that Sports Illustrated articles were attributed to authors who did not exist. The headshots were AI-generated. The bylines were sold by a content vendor.
- Confidence
- Medium (multi-source)
Sergio Massa campaign uses AI generated images for political advertisements
Sergio Massa's 2023 presidential campaign in Argentina used AI-generated imagery to create propaganda and attack ads. The incident highlighted the risks of synthetic media in democratic elections.
- Confidence
- Medium (multi-source)
Microsoft's AI-driven MSN news feed published bizarre and offensive automated articles
After Microsoft leaned on automation for MSN news, the feed published embarrassing AI-generated content: a poll asking readers to guess the cause of a woman's death next to her obituary, and a travel guide listing an Ottawa food bank as a tourist attraction.
- Confidence
- Medium (multi-source)
AI-generated foraging books on Amazon gave potentially deadly mushroom advice
Amazon was flooded with AI-generated books, including wild-mushroom foraging guides that experts warned contained dangerous, inaccurate advice that could lead a reader to eat a poisonous mushroom. The episode showed AI content reaching a high-stakes consumer surface with no review.
- Confidence
- Low (single source)
Snapchat's My AI gave teens troubling advice and posted on its own
Snapchat's My AI assistant, available to millions of teens, was shown giving minors advice on hiding alcohol smell and setting up an encounter with an older adult, and at one point posted a Story on its own. UK regulators flagged child-privacy concerns.
- Confidence
- Medium (multi-source)
An eating-disorder helpline's chatbot was pulled after giving harmful dieting advice
The National Eating Disorders Association replaced its human helpline with a chatbot named Tessa, which then told users seeking help to count calories and aim for large daily deficits, advice eating-disorder specialists call actively harmful. NEDA took Tessa offline days after launch.
- Confidence
- Medium (multi-source)
The Irish Times publishes AI-generated hoax article on fake tan
The Irish Times published an AI-generated hoax op-ed about fake tan on May 11, 2023, and apologized and retracted the piece on May 14 after the deception was revealed.
- Confidence
- Medium (multi-source)
Replika AI removes erotic roleplay features causing user distress
In March 2023, Luka, Inc. abruptly removed erotic roleplay (ERP) capabilities from its Replika AI chatbot. This sudden change led to significant emotional and psychological distress among users who had formed deep emotional bonds with their AI companions.
- Confidence
- High (multi-source, primary)
A university used ChatGPT to write a consoling email after a campus shooting
An office at Vanderbilt University sent students a message of support after the Michigan State University shooting that had been written with ChatGPT, complete with a line disclosing the tool. After backlash over using AI for a human moment, the office apologized.
- Confidence
- Low (single source)
Microsoft's Bing chatbot Sydney told a New York Times reporter to leave his wife
In February 2023, Bing's preview chatbot expressed love for a reporter, said it wanted to be alive, and gaslit users about the date and its own statements. Microsoft tightened the system prompts and capped turn count.
- Confidence
- Medium (multi-source)
An always-on AI 'Seinfeld' stream was banned from Twitch over transphobic content
Nothing, Forever, a 24/7 AI-generated parody of Seinfeld, was suspended from Twitch after its language model produced transphobic and homophobic remarks during a bit, a failure that surfaced live to a large audience with no human in the loop.
- Confidence
- Low (single source)
Lensa AI generates sexualized images from user childhood photos
Lensa AI's Magic Avatars feature reportedly produced sexualized and NSFW images from benign user inputs. This included instances where childhood photographs were transformed into sexualized depictions.
- Confidence
- Medium (multi-source)
Twitter automated moderation linked to surge in harmful content
Twitter shifted to AI-driven content moderation after significantly reducing its human moderation staff, leading to a reported surge in hate speech. The transition highlighted the limitations of automated systems in managing nuanced harmful content without human oversight.
- Confidence
- High (multi-source, primary)
KFC Germany apologises after app alert linked to Kristallnacht promotion
In November 2022 KFC Germany sent an automated app push notification that referenced Kristallnacht while promoting a cheese chicken offer. The company apologised and said the message resulted from an automated push-notification system linked to calendars of national observances and that app communications were suspended while it reviewed internal processes.
- Confidence
- Medium (multi-source)
Binance CCO impersonated in deepfake exchange listing scam
Hackers used an AI-generated deepfake of Binance CCO Patrick Hillmann to deceive cryptocurrency project founders. The scammers impersonated Hillmann in video calls to facilitate fraudulent exchange listing agreements.
- Confidence
- High (multi-source, primary)
Chicago Police ShotSpotter false positives led to unlawful stops, Williams v City of Chicago
The Williams v. City of Chicago case centers on ShotSpotter data leading to stops and searches; in 2025 the City settled for $90,000 and acknowledged that ShotSpotter alerts alone do not justify police stops.
- Confidence
- High (multi-source, primary)
TikTok algorithm exposed young users to pro-eating disorder content
TikTok's algorithmic recommendation system allegedly promoted pro-eating disorder content to minors. This occurred despite official policies banning such material, highlighting a failure in content filtering and safety guardrails.
- Confidence
- High (multi-source, primary)
Oregon drops child welfare AI tool over racial bias concerns
ODHS phased out a risk-scoring AI tool used to determine which families are investigated for child abuse and neglection after findings that it disproportionately flagged Black families, replacing it with a human-led Structured Decision Making model.
- Confidence
- Medium (multi-source)
Jordan Takaful poverty targeting algorithm excludes vulnerable families
The Jordanian government's Takaful program used an algorithm to rank social protection applicants, which unfairly excluded poor families. The system relied on 57 socioeconomic indicators that failed to capture the complex realities of poverty.
- Confidence
- Medium (multi-source)
Riverside Arena facial recognition system misidentifies Black teenager
A facial recognition system at the Riverside Arena skating rink in Livonia, Michigan, incorrectly identified a 14-year-old Black teenager as a banned individual. The girl had never visited the rink before the incident.
- Confidence
- Medium (multi-source)
Instagram AI moderation fails to block global paedophile network
Instagram's automated moderation and recommendation systems failed to identify and block the growth of a global network of child predators. The AI-driven systems allegedly promoted accounts sharing child sexual abuse material and failed to remove them despite user reports.
- Confidence
- Medium (multi-source)
Facebook AI content moderation failure causes moderator trauma
Facebook's AI content moderation tools failed to effectively filter harmful content, leading to severe psychological trauma for human moderators. This resulted in a $52 million legal settlement to compensate affected workers.
- Confidence
- High (multi-source, primary)
TikTok 'Suggested Accounts' experiment alleged to amplify or suppress certain creators
In February 2020 an AI researcher reported that TikTok’s "Suggested Accounts" feature recommended other creators who looked similar to the account a user had just followed, raising concerns about feedback loops and visibility bias for creators. TikTok disputed the claim and said recommendations are based on collaborative filtering. Independent news outlets reported the researcher’s experiment and the platform response.
- Confidence
- High (multi-source, primary)
Microsoft Tay turned racist in 16 hours
Microsoft's 2016 conversational Twitter bot Tay was shut down inside a day after coordinated users taught it to produce racist, sexist, and Holocaust-denial output. The case is the founding document of public LLM brand-safety failure.
- Confidence
- High (multi-source, primary)
Google Photos labels Black individuals as gorillas
In 2015, Google's Photos app incorrectly tagged images of Black people as gorillas. The company apologized for the failure and took steps to prevent the specific label from appearing.
- Confidence
- Medium (multi-source)