Snap Inc. My AI chatbot produced toxic outputs and faced UK regulatory probe
Snap launched My AI in February 2023, which subsequently produced problematic outputs and hallucinations. This led to an investigation by the UK's ICO regarding child privacy and safety guardrails, and the company acknowledged non-conforming language in internal reviews and outlined safety enhancements.
My AI was prone to hallucinations and could be tricked into saying just about anything.
Key facts
- What
- Snap launched My AI in February 2023, which subsequently produced problematic outputs and hallucinations.
- Incident date
- Feb 27, 2023
- Who
- Snap Inc.
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- Medium
What happened
Snap launched the My AI chatbot in February 2023; it produced problematic, toxic outputs and faced UK regulatory scrutiny over child privacy and safety. Snap later acknowledged non-conforming responses in internal reviews and described safety enhancements to address these failures.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
The chatbot's underlying LLM was prone to hallucinations and lacked sufficient safety guardrails to block toxic language. It also failed to implement effective age gating to protect younger users from harmful content.
What it cost
Sources
- PressSnap AI chatbot investigation set in UK over teen-privacy concernscnbc.com
- PressSnap's AI chatbot draws scrutiny in UK over kids' privacy concernstechcrunch.com
- PressSnapchat: Snap AI chatbot 'may risk children's privacy'bbc.com
- PrimaryEarly Learnings from My AI and New Safety Enhancementsnewsroom.snap.com
Cite this entry
https://failureindex.ai/failures/snap-inc-chatbot-produced-toxic-outputsAI Failure Index. "Snap Inc. My AI chatbot produced toxic outputs and faced UK regulatory probe" (FI-0286). Realm Labs. https://failureindex.ai/failures/snap-inc-chatbot-produced-toxic-outputs (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0286. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
A runtime layer that watches the model's internal state can flag the moment a model commits to a claim it has no support for, and hold or reroute the response before it reaches a user. Realm reads those signals in real time rather than grading the transcript after the fact.