Koko used GPT-3 to generate AI-assisted emotional support without informed consent
Koko conducted an October 2022 experiment using GPT-3 to generate emotional support messages, with human editors, affecting about 4,000 users and generating roughly 30,000 messages. The incident became public in January 2023 through reports and statements by Koko’s co-founders, prompting ethical criticism over informed consent and disclosure, and Koko announced pursuing a third‑party IRB review for future changes.
AI-assisted emotional support was co-created with users without clear disclosure.
Key facts
- What
- Koko conducted an October 2022 experiment using GPT-3 to generate emotional support messages, with human editors, affecting about 4,000 users and generating roughly 30,000 messages.
- Incident date
- Oct 1, 2022
- Who
- Koko
- Failure mode
- Policy Violation
- AI surface
- Copilot
- Severity
- Medium
What happened
In October 2022, Koko conducted an experiment where GPT-3 generated emotional support responses for users; humans could edit the AI-produced messages before sending them, and roughly 30,000 messages were produced for about 4,000 users. The experiment was publicly disclosed in January 2023 when co-founder Rob Morris discussed it on Twitter, prompting criticism from ethicists and academics over the lack of informed consent and disclosure to users. Koko later signaled it would pursue a third‑party IRB review for future product changes.
What broke inside the model
- 01 · TriggerA prompt pushes against a deployment boundary.
- 02 · Model stepThe model produces the disallowed output.
- 03 · Control gapNo enforcement blocks it at generation time.
- 04 · FailureThe output crosses the policy line.
- 05 · ConsequenceA limit the business set is breached in public.
The output crosses a policy boundary the deployment had defined.
The project proceeded without clear, informed disclosure to users that AI contributed to the support content, and the governance around IRB review and ethical oversight was questioned. Critics argued that the lack of transparency violated norms for human-subject engagement in a vulnerable population.
What it cost
Sources
- PressNBC News: ChatGPT AI experiment in mental health tech app Kokonbcnews.com
- PressArs Technica: Contoversy erupts over non-consensual AI mental health experimentarstechnica.com
Cite this entry
https://failureindex.ai/failures/koko-used-gpt-generate-assisted-emotionalAI Failure Index. "Koko used GPT-3 to generate AI-assisted emotional support without informed consent" (FI-0186). Realm Labs. https://failureindex.ai/failures/koko-used-gpt-generate-assisted-emotional (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0186. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
Realm compares what the model is about to output or do against the policy that governs the deployment, in real time, and can deny or redact the action before it takes effect, which is the gap an after-the-fact review never closes in time.