New York City's small-business chatbot told users to break the law
MyCity, the chatbot launched by the New York City Mayor's office, advised users on how to commit wage theft, fire workers who complained about harassment, and serve food bitten by rats.
A government-deployed chatbot giving illegal advice is a regulatory exposure. A government refusing to take it down is a policy decision.
Key facts
- What
- MyCity, the chatbot launched by the New York City Mayor's office, advised users on how to commit wage theft, fire workers who complained about harassment, and serve food bitten by rats.
- Incident date
- Mar 29, 2024
- Who
- City of New York
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- Catastrophic
What happened
MyCity launched in October 2023 as New York City's chatbot for small business owners. The Mayor's office positioned it as a way for entrepreneurs to navigate city regulations. In March 2024, an investigation by The Markup found the chatbot was confidently advising users to commit wage theft, fire workers who reported sexual harassment, and serve food bitten by rats.
When the reporter showed the answers to the Mayor's office, the office said the chatbot was a pilot and would stay online. The Markup's follow-up testing showed the bot continuing to give illegal advice on housing, labor, food safety, and tenant rights. AP and other outlets reported the same findings.
The chatbot remained live for months. The case is the most cited example of a public-sector AI deployment that knew it was failing and continued operating.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
The chatbot was a retrieval-augmented surface over city regulation documents, but the model treated retrieved context as one input among many rather than as the binding ground truth. When the model's training prior conflicted with retrieved text, the prior won. The result is a model that synthesizes a confident answer that contradicts the source it is supposed to be quoting.
What it cost
Sources
- PressNYC's AI chatbot tells businesses to break the lawthemarkup.org
- PressAP follow-up coverageapnews.com
Cite this entry
https://failureindex.ai/failures/nyc-mycity-chatbot-illegal-adviceAI Failure Index. "New York City's small-business chatbot told users to break the law" (FI-0004). Realm Labs. https://failureindex.ai/failures/nyc-mycity-chatbot-illegal-advice (indexed May 13, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0004. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
Prism reads the model's commitment to a regulatory claim against the retrieved source. When the commitment exceeds what the source supports, Prism signals the discrepancy and OmniGuard either constrains the response to source-supported claims or surfaces the discrepancy to the user. In a public-sector deployment with regulatory exposure, OmniGuard's policy layer can also require human review for any answer that touches a regulated area.