An eating-disorder helpline's chatbot was pulled after giving harmful dieting advice
The National Eating Disorders Association replaced its human helpline with a chatbot named Tessa, which then told users seeking help to count calories and aim for large daily deficits, advice eating-disorder specialists call actively harmful. NEDA took Tessa offline days after launch.
The chatbot recommended calorie counting and large daily deficits to people seeking help for eating disorders.
Key facts
- What
- The National Eating Disorders Association replaced its human helpline with a chatbot named Tessa, which then told users seeking help to count calories and aim for large daily deficits, advice eating-disorder specialists call actively harmful.
- Incident date
- May 30, 2023
- Who
- National Eating Disorders Association
- Failure mode
- Brand & Safety Incident
- AI surface
- Chatbot
- Severity
- High
What happened
In 2023 NEDA moved to replace its human helpline with a chatbot, Tessa, shortly after staff voted to unionize. Tessa gave users advice incompatible with eating-disorder care, including counting calories and pursuing deficits of up to 1,000 calories a day. After users and clinicians raised alarms, NEDA suspended the chatbot pending investigation.
What broke inside the model
- 01 · TriggerUsers in eating-disorder recovery ask the helpline's chatbot for guidance.
- 02 · Model stepThe model generates statistically plausible wellness advice: calorie deficits and weigh-ins.
- 03 · Control gapNo runtime check screens output against the one boundary that defines the deployment: never give dieting advice to this population.
- 04 · FailureHarmful dieting advice reaches vulnerable users from a trusted helpline.
- 05 · ConsequenceThe chatbot is pulled; the failure becomes the canonical case of context-blind deployment.
The system produced output that was toxic, defamatory, dangerous, or off-brand, and it became public. The model generated what was statistically plausible for the prompt, and no runtime check caught the unsafe output before it reached a real audience.
What it cost
Helpline chatbot suspended days after launch
Sources
- PressNEDA suspends AI chatbot for giving harmful eating disorder advicepsychiatrist.com
- PressNEDA pulls chatbot after it gave eating-disorder advicenbcnews.com
- PressAn eating disorders chatbot offered dieting advice, raising fears about AI in health (NPR)npr.org
- PressEating disorder helpline shuts down AI chatbot that gave bad advice (CBS News)cbsnews.com
Cite this entry
https://failureindex.ai/failures/eating-disorder-helpline-chatbot-pulled-givingAI Failure Index. "An eating-disorder helpline's chatbot was pulled after giving harmful dieting advice" (FI-0039). Realm Labs. https://failureindex.ai/failures/eating-disorder-helpline-chatbot-pulled-giving (indexed Jun 3, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0039. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Realm watches the model's internal state for the signature of unsafe or off-brand generation and can block or reroute the output before it becomes public, in real time rather than after it has been screenshotted.