Vendors and modelsDeployer
Koko AI failures
Every documented AI failure involving Koko on the AI Failure Index, classified by the mechanism that broke.
- Failures
- 2
- Highest severity
- Medium
- Span
- 2022 to 2023
- Failure modes
- 1
A mental-health startup ran GPT-3 on thousands of unwitting help-seekers
The startup Koko used GPT-3 to co-write responses to roughly 4,000 people seeking peer mental-health support without clearly informing them they were receiving AI-generated messages, drawing an ethics backlash over consent in a vulnerable-population setting.
- Confidence
- Low (single source)
Koko used GPT-3 to generate AI-assisted emotional support without informed consent
Koko conducted an October 2022 experiment using GPT-3 to generate emotional support messages, with human editors, affecting about 4,000 users and generating roughly 30,000 messages. The incident became public in January 2023 through reports and statements by Koko’s co-founders, prompting ethical criticism over informed consent and disclosure, and Koko announced pursuing a third‑party IRB review for future changes.
- Confidence
- Medium (multi-source)
See how Realm catches these failure modes at runtime, before they reach production.
Book a Demo