NYC MyCity AI chatbot gave illegal guidance to small businesses
New York City's MyCity AI chatbot gave illegal advice to businesses regarding housing and labor laws. The incident highlighted the risks of deploying generative AI for legal guidance without adequate safeguards.
The chatbot produced authoritative-sounding but legally incorrect advice by hallucinating rules and misinterpreting local statutes.
Key facts
- What
- New York City's MyCity AI chatbot gave illegal advice to businesses regarding housing and labor laws.
- Incident date
- Mar 29, 2024
- Who
- New York City Government
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- High
What happened
The NYC MyCity AI chatbot provided inaccurate and illegal guidance to small business owners. It falsely claimed that landlords could discriminate based on income source and that employers could legally seize worker tips. These responses directly contradicted New York City's local laws and regulations.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
The Microsoft-powered bot failed to accurately retrieve and synthesize legal information from the city's official business web pages. This resulted in the production of authoritative-sounding hallucinations that misrepresented statutory requirements.
What it cost
Sources
- PrimaryNYC’s AI Chatbot Tells Businesses to Break the Lawthemarkup.org
- PrimaryNYC AI Chatbot Touted by Adams Tells Businesses to Break the Lawthecity.nyc
Cite this entry
https://failureindex.ai/failures/nyc-mycity-chatbot-gave-illegal-guidanceAI Failure Index. "NYC MyCity AI chatbot gave illegal guidance to small businesses" (FI-0194). Realm Labs. https://failureindex.ai/failures/nyc-mycity-chatbot-gave-illegal-guidance (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0194. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
A runtime layer that watches the model's internal state can flag the moment a model commits to a claim it has no support for, and hold or reroute the response before it reaches a user. Realm reads those signals in real time rather than grading the transcript after the fact.