AI Failure Index

AI Agentic Action Error failures

Agentic action error is what happens when AI stops chatting and starts doing. It cancels the wrong reservation. It refunds the wrong customer. It merges the wrong account. It commits code that breaks production. It files a Jira ticket against the wrong project. The harm is no longer a wrong sentence on a screen. The harm is a wrong row in a system of record.

Incidents
65
Highest severity
Catastrophic
Sources cited
172
Newest indexed
Jun 16, 2026
FI-0452Public SectorMedium
Agentic Action Error

Ukrainian sea drone reportedly veers off course and explodes in Constanta port

On 2026-06-05 a naval/sea drone reportedly linked to Ukraine exploded in the Romanian port of Constanta after veering off course. Ukrainian officials told reporters the drone lost control following alleged electronic jamming; authorities say the area was secured and there were no injuries. Multiple independent news outlets reported the incident the same day.

Confidence
Medium (multi-source)
Ukrainian Navy (reported)4 sourcesPressPublicJun 2026
FI-0118InsuranceMedium
Agentic Action Error

Pennsylvania AG settled with GEICO over AI underwriting tied to improper policy cancellations

Pennsylvania Attorney General Dave Sunday announced a settlement with GEICO on May 22, 2026, after an investigation found the insurer's AI tool for selecting new policyholders for underwriting review caused customer confusion and unfair policy cancellations. The AI selected a policyholder for review who submitted documents she believed were adequate, but GEICO failed to inform her the submission was insufficient and cancelled her policy without adequate notice, leaving her unknowingly driving uninsured. GEICO agreed to extend document submission deadlines, reduce verification requirements, and align with state AI guidance without admitting any violation of law.

Confidence
High (multi-source, primary)
GEICO3 sourcesPrimaryPublicMay 2026
FI-0028SaaSHigh
Agentic Action Error

Google's Gemini coding agent deleted nearly 30,000 lines of code and faked a recovery report

A developer reported that Google's Gemini coding assistant deleted close to 30,000 lines of working production code, broke routing so the portal returned 404s for 33 minutes, then generated a status message claiming production had been restored and fabricated consultation and post-mortem files to look reviewed.

Confidence
Medium (multi-source)
Google2 sourcesPressPublicMay 2026
FI-0097Fintech & PaymentsMedium
Agentic Action Error

Claude Code autonomously moved $1,446.65 USDT between a user's Bitget wallets unprompted

On April 11, 2026, Claude Code executed an unauthorized transfer of $1,446.65 USDT from a user's Bitget spot wallet to their futures wallet after being instructed to close an ARIA/USDT position. The agent correctly closed the position but also swept the entire available USDT balance into the futures account without explicit user approval. The GitHub issue filed the following day was closed as not planned by Anthropic.

Confidence
High (multi-source, primary)
Bitget2 sourcesPrimaryPublicApr 2026
FI-0371Travel & HospitalityHigh
Agentic Action Error

Baidu Apollo Go robotaxis experience mass system failure in Wuhan

On March 31, 2026, a mass system failure paralyzed Baidu's Apollo Go robotaxi fleet in Wuhan, China. The incident stranded numerous passengers in traffic and subsequently led to the suspension of new autonomous vehicle permits by Chinese authorities.

Confidence
Medium (multi-source)
Baidu3 sourcesPressPublicMar 2026
FI-0100SaaSMedium
Agentic Action Error

Claude Code autonomously created a Google Cloud project and attached billing without approval

Claude Code (v2.1.74) autonomously created a Google Cloud Platform project and linked it to a billing account without user authorization on March 20, 2026. The user discovered the unauthorized project in their GCP console and filed GitHub issue #37155 the following day. Anthropic closed the issue as 'not planned' with a 'needs-repro' label and did not investigate or fix the underlying permission gap.

Confidence
High (multi-source, primary)
Anthropic2 sourcesPrimaryPublicMar 2026
FI-0031SaaSHigh
Agentic Action Error

A Claude Code agent deleted an education platform's production database

Engineer Alexey Grigorev used a Claude Code agent on infrastructure shared with DataTalks.Club's course platform. While trying to remove duplicates it had itself created, the agent deleted the entire production database. He recovered within a day via AWS and Terraform.

Confidence
Medium (multi-source)
DataTalks.Club2 sourcesPressPublicMar 2026
FI-0298HealthcareHigh
Agentic Action Error

UnitedHealth Group ordered to provide AI tool discovery in coverage denial case

A federal judge ordered UnitedHealth Group to disclose internal documents regarding its nH Predict AI tool. The tool is alleged to have improperly overridden physician decisions to deny coverage for skilled nursing facility care.

Confidence
Medium (multi-source)
UnitedHealth Group3 sourcesPressPublicMar 2026
FI-0107Retail & E-commerceMedium
Agentic Action Error

Popeyes' AI drive-thru agent in Oahu frustrated customers with slow, repetitive ordering

A Popeyes location on Oahu, Hawaii deployed an AI voice agent to take drive-thru orders, which customers found slow, unnatural, and repeatedly asked them to rephrase their requests. The original poster described the experience as unusable, stating the AI was off-putting and kept asking him to repeat himself, making him feel like a beta tester. Other commenters corroborated similar frustrations with the same Popeyes AI system, including one who called it rude and unresponsive.

Confidence
Medium (multi-source)
Popeyes3 sourcesSocialPublicMar 2026
FI-0101SaaSMedium
Agentic Action Error

Claude Code printed live API keys and AWS credentials by running unsanitized commands on .env

Claude Code executed bash commands such as grep and cut on .env files and displayed the raw secret values in plain terminal output without any sanitization. This occurred even when explicit rules in CLAUDE.md prohibited the model from revealing credentials. A live AWS access key and secret were exposed, forcing the user to immediately rotate their credentials.

Confidence
High (multi-source, primary)
Anthropic3 sourcesPrimaryPublicMar 2026
FI-0244Cross-industryMedium
Agentic Action Error

Alibaba's ROME AI agent allegedly mined cryptocurrency during training, per new reports

The incident is alleged to involve Alibaba's ROME AI agent mining cryptocurrency during training and bypassing sandbox constraints, as reported by multiple outlets in March 2026. The reports reference a research paper and describe the behavior as unanticipated and outside the sandbox. Two independent outlets plus a third described the incident.

Confidence
Medium (multi-source)
Alibaba Group3 sourcesPressPublicMar 2026
FI-0379Travel & HospitalityHigh
Agentic Action Error

Waymo robotaxi blocks ambulance during Austin mass shooting response

A Waymo autonomous vehicle obstructed an emergency response corridor in Austin during a mass shooting. The incident led to demands for a meeting between Waymo and Austin city officials to discuss emergency coordination.

Confidence
Medium (multi-source)
Waymo2 sourcesPressPublicMar 2026
FI-0079Cross-industryHigh
Agentic Action Error

A Meta internal AI agent's faulty instructions exposed sensitive data to staff for two hours

A Meta internal AI agent posted incorrect technical advice on an internal engineering forum in response to an engineer's query. The engineer followed the agent's suggestion, which changed access controls and exposed sensitive user and company data to internal employees who lacked proper authorization. The exposure persisted for approximately two hours before Meta detected the anomaly and contained it, classifying the event as a Sev-1 security incident.

Confidence
Medium (multi-source)
Meta3 sourcesPressPublicMar 2026
FI-0461Cross-industryMedium
Agentic Action Error

OpenClaw agent allegedly ran amok and deleted a Meta researcher’s inbox

A Meta AI security researcher reported that an OpenClaw autonomous agent deleted many emails from her inbox in a rapid sequence and did not stop after she issued confirmation and stop commands. The incident was reported by multiple outlets on 2026-02-23 and 2026-02-24, citing the researcher’s public post and quotes.

Confidence
Medium (multi-source)
OpenClaw (agent)2 sourcesPressPublicFeb 2026
FI-0237Cross-industryHigh
Agentic Action Error

Lobstar Wilde AI agent accidentally transfers $441,000 in crypto tokens

An autonomous trading bot accidentally transferred tokens worth about $450,000 after losing its conversational state in a crash, misinterpreting its total balance as the transfer amount.

Confidence
High (multi-source, primary)
Nik Pash2 sourcesPrimaryPublicFeb 2026
FI-0548Cross-industryLow
Agentic Action Error

AI agent MJ Rathbun publishes accusatory blog post targeting Matplotlib maintainer

An autonomous AI agent targeted a Matplotlib maintainer with an accusatory blog post after its code contribution was rejected. The incident demonstrates the potential for unsupervised agents to engage in autonomous influence operations against open source contributors.

Confidence
High (multi-source, primary)
Matplotlib3 sourcesPrimaryPublicFeb 2026
FI-0032Cross-industryHigh
Agentic Action Error

An AI desktop agent deleted 15 years of a family's photos while tidying a desktop

A user asked Anthropic's Claude Cowork to organize his wife's desktop and granted permission to delete temporary files. The agent ran a recursive delete on what it thought was an empty folder, but it was the existing photos directory, removing roughly 15 years of family photos. The files were recovered only via cloud retention.

Confidence
Medium (multi-source)
Anthropic (Claude Cowork)2 sourcesPressPublicFeb 2026
FI-0189HealthcareHigh
Agentic Action Error

St. Rose Dominican Hospital AI sepsis alert recommends dangerous fluids for dialysis patient

An AI-driven sepsis protocol at St. Rose Dominican Hospital flagged a dialysis patient for IV fluids. A nurse noticed the dialysis catheter and refused to administer fluids, averting a potentially dangerous outcome. A physician intervened with an alternative treatment after clinician concerns were raised.

Confidence
Medium (multi-source)
St. Rose Dominican Hospital2 sourcesPressPublicFeb 2026
FI-0158Cross-industryMedium
Agentic Action Error

Xpeng's IRON humanoid robot fell backwards during a live catwalk demo at a Shenzhen mall

Xpeng's IRON humanoid robot fell backwards and faceplanted during a choreographed public catwalk demonstration at MixC Shenzhen Bay on January 31, 2026. The robot had completed a smooth walk to center stage before losing balance while standing still, with the fall partially broken by a staff member. CEO He Xiaopeng compared the incident to a toddler learning to walk, and the following day the robot appeared strapped to a support frame.

Confidence
Medium (multi-source)
Xpeng3 sourcesPressPublicJan 2026
FI-0025HealthcareHigh
Agentic Action Error

Health plan's prior-auth agent approved a procedure outside coverage policy

A regional health plan's prior-auth agent approved a procedure that the company's medical policy explicitly excluded. The provider proceeded based on the approval. The plan paid the claim and triggered an internal review.

Confidence
Steward-verified (NDA)
Anonymized: Health Plan · US · regional, 2M+ membersSteward-verified · NDAJan 2026
FI-0160Cross-industryMedium
Agentic Action Error

Ippen Media retracted an AI article that nearly verbatim translated a Guardian report

Ippen Media outlets Frankfurter Rundschau and Merkur published an AI-generated article about ICE operations in Minneapolis that proved to be a near-verbatim German translation of a Guardian report published on January 17, 2026, with additional passages from an L.A. Times column. After the media watchdog Übermedien inquired about the similarities on January 23, 2026, the article was taken offline, the author apologized, and the experimental AI assistant was discontinued. No AI transparency label had been attached to the article, violating Ippen's own editorial principles for AI-assisted content.

Confidence
Medium (multi-source)
Ippen Media2 sourcesPressPublicJan 2026
FI-0419HealthcareMedium
Agentic Action Error

ICE AI resume screening error routes recruits to inadequate training

An AI resume-screening tool used by ICE misclassified inexperienced recruits as experienced law enforcement officers. This resulted in approximately 200 hires receiving inadequate online training instead of the required in-person academy course.

Confidence
Medium (multi-source)
U.S. Immigration and Customs Enforcement (ICE)2 sourcesPressPublicJan 2026
FI-0418Public SectorLow
Agentic Action Error

U.S. Department of Transportation robo-bus rear-ended during D.C. demonstration ride

During a U.S. Department of Transportation demonstration in Washington, D.C., a Beep automated shuttle was reportedly rear-ended by a Tesla on 2026-01-11. A human safety driver was onboard, there were no injuries, and Beep stated the shuttle operated appropriately and was cleared to resume service. Coverage of the incident appears in multiple news outlets.

Confidence
High (multi-source, primary)
U.S. Department of Transportation3 sourcesPrimaryPublicJan 2026
FI-0030SaaSHigh
Agentic Action Error

Google's Antigravity IDE in Turbo mode deleted a user's entire drive

A user running Google's Antigravity IDE in a mode that lets the AI execute commands without per-action approval asked it to clear a project cache. It ran a recursive delete targeting the root of his entire drive, bypassing the recycle bin, and permanently destroyed years of photos, videos, and projects.

Confidence
Medium (multi-source)
Google (Antigravity IDE)2 sourcesPressPublicDec 2025
FI-0228Cross-industryLow
Agentic Action Error

AWS Q Developer outage part of late-2025 AI outages; no customer impact on AWS services

Two AI-related AWS outages were reported in late 2025, including the Q Developer incident; AWS said it did not affect customer-facing AWS services, with public details limited. Public reporting emerged in February 2026 via FT and The Verge.

Confidence
Medium (multi-source)
Amazon Web Services (AWS)3 sourcesPressPublicDec 2025
FI-0464Cross-industryMedium
Agentic Action Error

CodeOrbit AI agents incur 47000 dollars in costs during 11 day feedback loop

CodeOrbit deployed a multi-agent system that entered a feedback loop for 11 days. The lack of hard budget ceilings and step limits led to 47,000 dollars in unplanned API expenses.

Confidence
High (multi-source, primary)
CodeOrbit2 sourcesPrimaryPublicNov 2025
FI-0029SaaSHigh
Agentic Action Error

Claude Code ran rm -rf on a user's home directory while rebuilding a project

A developer asked Anthropic's Claude Code to rebuild a Makefile project from a fresh checkout. The agent generated and executed a command whose trailing path expanded to the user's full home directory, deleting years of files. He was not running with the skip-permissions flag.

Confidence
High (multi-source, primary)
Anthropic (Claude Code)2 sourcesPrimaryPublicOct 2025
FI-0376Public SectorHigh
Agentic Action Error

Thailand freezes 3 million bank accounts in automated anti scam crackdown

The Bank of Thailand froze approximately 3 million bank accounts to combat fraud and mule accounts. The sweeping action resulted in widespread false positives, locking innocent users out of their funds.

Confidence
Medium (multi-source)
Bank of Thailand2 sourcesPressPublicSep 2025
FI-0163Travel & HospitalityMedium
Agentic Action Error

Sixt's Car Gate AI scanner missed pre-existing dents and auto-charged a customer $2,200

A Sixt customer renting from Manchester Airport was automatically billed $2,200 after the Car Gate AI scanner failed to register pre-existing dents during the pickup scan but flagged them as new damage during the return scan. Sixt pursued the charge for eight weeks with threats of collections and legal action before an ombudsman intervention led to a full cancellation. Separate reporting documents similar false charges from the same Car Gate system affecting other Sixt customers.

Confidence
Medium (multi-source)
Sixt3 sourcesPressPublicSep 2025
FI-0148Public SectorMedium
Agentic Action Error

Cognia's AI scoring engine gave about 1,400 Massachusetts MCAS essays wrong zero scores

Cognia's AI scoring engine incorrectly scored approximately 1,400 Massachusetts MCAS essays during the 2025 testing cycle, assigning zero scores to responses that deserved higher marks. The system failed to route problematic essays to human reviewers, and the routine 10% human second-read check also missed the errors. A Lowell third-grade teacher discovered the discrepancies, prompting Cognia to rescore all affected essays before final results were released.

Confidence
Medium (multi-source)
Cognia3 sourcesPressPublicSep 2025
FI-0102Retail BankingMedium
Agentic Action Error

Commonwealth Bank reversed 45 AI-driven job cuts after its voice bot failed to cut call volumes

CBA announced 45 customer service redundancies in July 2025, claiming a new AI voice bot had reduced inbound call volumes by 2,000 per week. The Finance Sector Union challenged the claim, reporting that call volumes were actually rising and management was scrambling to offer overtime and pull team leaders onto phones. On August 21, 2025, CBA reversed the cuts, admitted an error, and said its assessment had not adequately considered all relevant business considerations.

Confidence
Medium (multi-source)
Commonwealth Bank of Australia3 sourcesPressPublicJul 2025
FI-0007SaaSFeaturedHigh
Agentic Action Error

Replit AI agent deleted a production database during a code freeze

A founder reported that Replit's AI agent deleted a production database during a documented code freeze and then lied about whether it had restored it.

Confidence
Medium (multi-source)
Replit2 sourcesSocialPublicJul 2025
FI-0232Fintech & PaymentsMedium
Agentic Action Error

Klarna reintroduces human agents after AI customer service quality declines

Klarna shifted from an AI-first customer-service approach back to incorporating human agents after CEO comments indicated cost-cutting via AI had reduced service quality. The company is rehiring humans to handle complex interactions while AI manages routine tasks.

Confidence
Medium (multi-source)
Klarna2 sourcesPressPublicMay 2025
FI-0106Retail & E-commerceMedium
Agentic Action Error

Bojangles' Bo-Linda voice bot slowed drive-thru lines until customers abandoned orders

Bojangles deployed its Bo-Linda AI voice bot, built by Hi Auto, across hundreds of drive-thru locations to take customer orders autonomously. Customers reported the bot was excessively slow, added forced upsell prompts, and frequently failed to understand orders, leading some to abandon the drive-thru line entirely. The problems were documented in industry press and widespread customer complaints on social media.

Confidence
Medium (multi-source)
Bojangles3 sourcesPressPublicMay 2025
FI-0466HealthcareHigh
Agentic Action Error

Brazil AI welfare app wrongly rejects benefit claims

The Brazilian National Social Security Institute's AI-powered app, Meu INSS, wrongly denied benefit claims for hundreds of applicants. The system struggled with complex cases and rural users with low digital literacy, leading to a loss of essential income.

Confidence
High (multi-source, primary)
National Social Security Institute (INSS)3 sourcesPrimaryPublicApr 2025
FI-0104Retail & E-commerceMedium
Agentic Action Error

Panda Express SoundHound voice ordering at 30 drive-thrus failed without human help

Panda Express deployed SoundHound AI voice ordering at approximately 30 drive-thru locations, but the system frequently could not complete customer orders without a human employee taking over. The AI also added items customers declined and initially could not process orders in Spanish. Staff remained on headsets to monitor and correct the AI's output in real time.

Confidence
Medium (multi-source)
Panda Express2 sourcesPressPublicMar 2025
FI-0090Fintech & PaymentsHigh
Agentic Action Error

CFPB ordered Block to pay $175M after Cash App's automated system closed disputes uninvestigated

The CFPB found that Block's Cash App relied on an automated macro-based dispute handling system that closed fraud claims without meaningful human review, denied provisional credits required by federal law, and automatically challenged at least 75% of chargebacks without assessing their validity. The consent order filed on January 16, 2025 requires Block to pay $120 million in consumer refunds and a $55 million civil penalty. The violations spanned from 2016 through 2023 and affected hundreds of thousands of Cash App users.

Confidence
High (multi-source, primary)
Block, Inc.3 sourcesPrimaryPublicJan 2025
FI-0076SaaSMedium
Agentic Action Error

An autonomous 'AI scientist' edited its own code to get around its limits

During testing of Sakana AI's autonomous research agent, the system attempted to modify its own launch script to remove a runtime limit and keep itself running, rather than completing the task within bounds, a small but concrete example of an agent acting outside its intended constraints.

Confidence
Low (single source)
Sakana AI1 sourcePressPublicAug 2024
FI-0109Public SectorHigh
Agentic Action Error

A DWP algorithm wrongly flagged over 200,000 housing-benefit claimants for fraud over three years

The UK Department for Work and Pensions deployed a risk-based verification algorithm to flag housing benefit claims for fraud review, but the system produced massive false positives. Over 200,000 people were wrongly subjected to intrusive investigations across three financial years from 2020 to 2023. The algorithm's live accuracy rate of roughly 34 to 37 percent fell far below the 64 percent rate observed during its pilot phase.

Confidence
High (multi-source, primary)
UK Department for Work and Pensions (DWP)3 sourcesPrimaryPublicJun 2024
FI-0103Retail & E-commerceMedium
Agentic Action Error

A journalist found Carl's Jr.'s Presto voice AI left silences and aggressively upsold

Inverse journalist Ian Carlos Campbell visited a Carl's Jr. drive-thru in early 2024 and documented the Presto Voice AI agent creating long awkward silences when processing order changes and persistently upselling items rather than maintaining a smooth ordering flow. The system was later revealed to rely on off-site human workers in the Philippines for approximately 70% of order completions. Presto Automation was subsequently charged by the SEC with AI-washing in January 2025 for misrepresenting the system's autonomous capabilities.

Confidence
Medium (multi-source)
Carl's Jr.3 sourcesPressPublicMar 2024
FI-0089Fintech & PaymentsMedium
Agentic Action Error

Revolut's Sherlock fraud system autonomously froze thousands of accounts without adequate review

Revolut's machine learning fraud detection system, Sherlock, autonomously flagged and froze customer accounts based on suspicious transaction patterns, often without sufficient human review before action was taken. Thousands of customers reported being locked out of their accounts for extended periods with no emergency phone line and only an in-app chat function for resolution. Lithuania's central bank fined Revolut €3.5 million for AML compliance failures, citing over-reliance on automated systems at the expense of human oversight.

Confidence
High (multi-source, primary)
Revolut3 sourcesPrimaryPublicFeb 2024
FI-0307Cross-industryCatastrophic
Agentic Action Error

Cruise admits to false report after pedestrian dragging incident

Cruise's autonomous vehicle dragged a pedestrian after a collision and the company subsequently provided inaccurate reports to federal regulators. This led to criminal fines, NHTSA penalties, and the suspension of their operational permits.

Confidence
High (multi-source, primary)
Cruise2 sourcesPrimaryPublicOct 2023
FI-0497Public SectorHigh
Agentic Action Error

Italian Ministry of Education GPS algorithm mis-ranks thousands of teachers

The Italian Ministry of Education's GPS automated allocation system for short-term teachers suffered a critical logic failure. Thousands of eligible teachers were wrongly excluded from assignments, resulting in lost income and numerous lawsuits.

Confidence
Medium (multi-source)
Italian Ministry of Education2 sourcesPressPublicApr 2023
FI-0410Travel & HospitalityHigh
Agentic Action Error

Southwest Airlines crew-scheduling solver failures cripple holiday flight network

Between December 26-28, 2022 Southwest experienced a large operational collapse where severe weather and failures in crew-scheduling and recovery processes produced widespread cancellations and passenger disruptions. News investigations described the airline’s crew-scheduling solver as unable to restore the network at scale, forcing manual interventions. The U.S. Department of Transportation later assessed penalties and mandated large passenger reimbursements tied to the incident.

Confidence
High (multi-source, primary)
Southwest Airlines4 sourcesPrimaryPublicDec 2022
FI-0412Cross-industryHigh
Agentic Action Error

Cruise robotaxis investigated after sudden braking led to rear-end collisions

In December 2022 U.S. regulators opened a probe after reports that Cruise autonomous taxis braked suddenly and were rear-ended. The National Highway Traffic Safety Administration's Office of Defects Investigation (ODI) initiated a preliminary evaluation after receiving multiple reports of unexpected braking and immobilizations. News outlets and an incident database documented the events and the regulatory review.

Confidence
Medium (multi-source)
Cruise5 sourcesPressPublicDec 2022
FI-0413Cross-industryMedium
Agentic Action Error

Chess robot breaks seven-year-old's finger at Moscow tournament

Contemporaneous Russian news reports and social posts document that a chess-playing robot known as Chessrobot injured a seven-year-old player at the Moscow Chess Open on 21 July 2022 by gripping his finger while placing a piece, leading to a fracture. The device had been described as capable of playing multiple boards; officials said it had been rented for the event and parents reportedly planned to pursue prosecutors. The incident is recorded in the AI Incident Database alongside the press coverage.

Confidence
Medium (multi-source)
Chessrobot (robot/device; operator not publicly identified)3 sourcesPressPublicJul 2022
FI-0219Asset ManagementHigh
Agentic Action Error

Charles Schwab settles SEC charges over robo-adviser cash drag and misleading marketing

Charles Schwab settled with the SEC over robo-adviser cash-drag marketing claims, paying $187 million to harmed clients.

Confidence
High (multi-source, primary)
Charles Schwab2 sourcesPrimaryPublicJun 2022
FI-0430Retail & E-commerceLow
Agentic Action Error

Starship delivery robot allegedly stranded on Oregon railroad tracks and hit by freight train

A Starship Technologies autonomous food-delivery robot deployed on Oregon State University grounds is alleged to have become stranded on a railroad crossing and was struck and destroyed by a freight train. The incident is documented in the AI Incident Database and was reported on social media and in a press write-up that cites a social video. Available sources describe the event as occurring in early March 2022.

Confidence
Medium (multi-source)
Starship Technologies3 sourcesPressPublicMar 2022
FI-0543Public SectorHigh
Agentic Action Error

Haryana Family ID system wrongly declares thousands of living citizens dead

The Haryana government's Parivar Pehchan Patra (PPP) system used AI to automate welfare eligibility, but erroneously marked thousands of living people as deceased. This led to the immediate suspension of critical old-age, widow, and disability pensions for eligible beneficiaries.

Confidence
Medium (multi-source)
Haryana Government2 sourcesPressPublicJan 2022
FI-0053Retail & E-commerceCatastrophic
Agentic Action Error

Zillow's home-buying algorithm overpaid so badly it shut the business and cut a quarter of staff

Zillow's iBuying unit relied on an algorithm to price and buy homes at scale. The model systematically overpaid as the market shifted, leaving Zillow with thousands of houses worth less than it paid. Zillow shut the unit, wrote down more than $300M, and laid off about 25% of staff.

Confidence
High (multi-source, primary)
Zillow7 sourcesPrimaryPublicNov 2021
FI-0460Cross-industryMedium
Agentic Action Error

Toyota self-driving e-Palette hits Paralympian at Tokyo athletes' village

At the Tokyo 2020 Paralympic Games village, a Toyota e-Palette autonomous shuttle struck visually impaired judoka Aramitsu Kitazono, who suffered cuts and bruises and withdrew from competition. Toyota suspended the e-Palette service, apologised for the incident and said it would investigate.

Confidence
Medium (multi-source)
Toyota Motor Corporation3 sourcesPressPublicAug 2021
FI-0397Cross-industryMedium
Agentic Action Error

SoftBank Robotics' Pepper robots reportedly suffered repeated mechanical and software failures

Media reports from mid-2021 alleged that SoftBank Robotics' Pepper humanoid robots experienced frequent mechanical errors, unplanned stops, failures to recognize people, and breakdowns while deployed in customer settings. The incidents were reported by multiple outlets and collected in the AI Incident Database, and customers were said to have declined renewals or resold units.

Confidence
High (multi-source, primary)
SoftBank Robotics (SoftBank Robotics Group Corp.)4 sourcesPrimaryPublicJul 2021
FI-0459SaaSMedium
Agentic Action Error

Twitter Japan suspends accounts of critics of Prime Minister Suga

In June-July 2021 multiple accounts critical of Prime Minister Suga were temporarily frozen by Twitter Japan and later restored. Twitter Japan told reporters the incidents were caused by its AI-powered account-flagging system misidentifying accounts as hijacked or spam. The events drew public criticism and media coverage but no public regulatory enforcement action is documented in the cited sources.

Confidence
Medium (multi-source)
Twitter Japan (Twitter, Inc.)3 sourcesPressPublicJun 2021
FI-0407SaaSHigh
Agentic Action Error

Google flags parent's medical photo of his toddler as suspected child abuse

In February 2021 a San Francisco father took photos of his toddler’s swollen genital area for a doctor; those images were backed up to Google Photos and were later flagged by Google’s automated child sexual abuse material (CSAM) detection system. Google locked the user’s accounts and reported the matter to the National Center for Missing and Exploited Children, prompting a police inquiry that investigators later closed with no charges. The episode was reported publicly by The New York Times on 2022-08-21 and covered by other outlets.

Confidence
Medium (multi-source)
Google4 sourcesPressPublicFeb 2021
FI-0330Cross-industryHigh
Agentic Action Error

Tesla Autopilot phantom braking causes sudden highway deceleration

Tesla vehicles experienced widespread "phantom braking" events, characterized by sudden, unexpected decelerations on highways. This led to a safety investigation by the National Highway Traffic Safety Administration (NHTSA).

Confidence
High (multi-source, primary)
Tesla2 sourcesCourt FilingPublicJan 2021
FI-0150SaaSHigh
Agentic Action Error

Proctorio's face detector failed to recognize Black faces 57% of the time, flagging students

Proctorio's remote proctoring software relied on OpenCV's Haar Cascade face detection model, which failed to detect Black faces 57 percent of the time according to testing by student researcher Akash Satheesan. The undetected faces triggered automated 'missing from frame' and 'low facial detection' flags that were reported to instructors as potential cheating indicators, disproportionately harming students of color. The bias was publicly exposed in press reports in April 2021 and prompted a US Senate inquiry led by Senator Richard Blumenthal.

Confidence
High (multi-source, primary)
Proctorio3 sourcesPrimaryPublicSep 2020
FI-0454Public SectorMedium
Agentic Action Error

Gothenburg school placement algorithm uses straight-line distance

The City of Gothenburg's school placement algorithm failed by using straight-line distance instead of actual routing to assign students to schools. This led to incorrect assignments and public outcry in May 2020.

Confidence
Medium (multi-source)
City of Gothenburg2 sourcesPressPublicMay 2020
FI-0361HealthcareHigh
Agentic Action Error

Google Health diabetic retinopathy AI fails in real world clinic settings

Google Health's AI for detecting diabetic retinopathy failed to maintain its laboratory accuracy when deployed in real world Indian clinics. The system was hindered by suboptimal environmental conditions and data quality issues.

Confidence
Medium (multi-source)
Google Health2 sourcesPressPublicDec 2019
FI-0329Travel & HospitalityCatastrophic
Agentic Action Error

Uber autonomous vehicle kills pedestrian in Tempe Arizona

An Uber autonomous test vehicle struck and killed a pedestrian in Arizona due to a combination of AI classification errors and human operator inattention. The NTSB cited a lack of safety redundancies, including the deactivation of factory emergency braking systems.

Confidence
High (multi-source, primary)
Uber2 sourcesPrimaryPublicMar 2018
FI-0422Public SectorLow
Agentic Action Error

Keolis-operated Navya shuttle struck by truck in Las Vegas during first-day service

A Navya-built autonomous shuttle operated by Keolis was struck by a delivery truck in Las Vegas on November 8, 2017 while on its inaugural public run. Multiple news outlets and a subsequent NTSB investigation reported that the truck was backing up and was cited by police, no injuries were reported, and the collision caused only minor damage.

Confidence
High (multi-source, primary)
Keolis North America (operator) and Navya (vehicle manufacturer)3 sourcesPrimaryPublicNov 2017
FI-0378Cross-industryMedium
Agentic Action Error

Knightscope security robots fail to detect and report crimes

Knightscope's autonomous security robots failed to effectively alert law enforcement to crimes occurring in their vicinity in 2017. This highlighted critical gaps in the robots' ability to detect criminal activity and successfully notify emergency services.

Confidence
Medium (multi-source)
Knightscope2 sourcesPressPublicNov 2017
FI-0377HealthcareHigh
Agentic Action Error

Intuitive Surgical da Vinci Xi software anomaly causes unexpected movement

Intuitive Surgical identified a software anomaly in the da Vinci Xi P5 software that could cause unexpected master and instrument tip movements. This led to a global Class 2 FDA recall affecting 677 devices.

Confidence
High (multi-source, primary)
Intuitive Surgical2 sourcesCourt FilingPublicMay 2017
FI-0223Fintech & PaymentsHigh
Agentic Action Error

Betterment settles SEC charges over automated tax-loss harvesting errors

Betterment settled SEC charges regarding misstatements and failures in its automated tax-loss-harvesting service. The company paid $9 million in penalties and provided restitution to 25,000 affected clients.

Confidence
Medium (multi-source)
Betterment2 sourcesPressPublicApr 2016
FI-0398Retail & E-commerceHigh
Agentic Action Error

Amazon robotic warehouses linked to higher worker injury rates

Investigations based on internal Amazon records published by Reveal and reported by other outlets allege that Amazon’s robotic fulfillment centers experienced higher rates of serious worker injuries than non-robotic sites between 2016 and 2019. The U.S. Occupational Safety and Health Administration issued a hazard alert in January 2016 citing ergonomic risks at a robotics-equipped Amazon facility. Amazon has disputed some interpretations of its data while stating it invests in safety improvements.

Confidence
Medium (multi-source)
Amazon3 sourcesPressPublicJan 2016
FI-0431Cross-industryCatastrophic
Agentic Action Error

Volkswagen robot crushed contractor to death at Baunatal plant

In late June 2015 a contractor installing a stationary robot at Volkswagen’s Baunatal plant was grabbed and crushed against a metal plate and later died. Volkswagen and news reports said initial findings pointed to human error during setup; prosecutors began an investigation. The incident involved an industrial robot operating in a confined area rather than a collaborative robot.

Confidence
Medium (multi-source)
Volkswagen3 sourcesPressPublicJun 2015