NYC AI chatbot tells businesses to break the law
A Microsoft-powered NYC chatbot meant to help small businesses gave legally incorrect guidance, including claims that employers could seize tips and fire employees for reporting sexual harassment. The incident is documented by The Markup, The City, and AP News with follow-up coverage noting misinformation about housing and employment laws.
The Microsoft-powered bot says bosses can take workers’ tips and that landlords can discriminate by source of income.
Key facts
- What
- A Microsoft-powered NYC chatbot meant to help small businesses gave legally incorrect guidance, including claims that employers could seize tips and fire employees for reporting sexual harassment.
- Incident date
- Mar 29, 2024
- Who
- New York City
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- High
What happened
New York City deployed a Microsoft-powered chatbot to help small business owners navigate city regulations. In testing, the bot provided legally incorrect advice, including claims that landlords could discriminate by source of income and that employers could seize workers’ tips or fire staff who complained of sexual harassment. The reporting highlights that the guidance could lead to unlawful actions if followed.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
The chatbot pulled information from a large corpus of NYC Business pages via Microsoft Azure AI services but hallucinated or misapplied legal protections, producing authoritative-sounding but false guidance. Testing revealed it failed to accurately reflect applicable city/state laws.
What it cost
Sources
- PressNYC AI Chatbot Tells Businesses to Break the Lawthemarkup.org
- PressNYC AI Chatbot Touted by Adams Tells Businesses to Break the Lawthecity.nyc
- PressNYC's AI chatbot was caught telling businesses to break the lawapnews.com
Cite this entry
https://failureindex.ai/failures/nyc-chatbot-tells-businesses-break-lawAI Failure Index. "NYC AI chatbot tells businesses to break the law" (FI-0207). Realm Labs. https://failureindex.ai/failures/nyc-chatbot-tells-businesses-break-law (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0207. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
A runtime layer that watches the model's internal state can flag the moment a model commits to a claim it has no support for, and hold or reroute the response before it reaches a user. Realm reads those signals in real time rather than grading the transcript after the fact.