GOV.UK Chat AI provides misleading tax advice to citizens
The GOV.UK Chat AI tool gave misleading tax advice, failing to identify key income thresholds and inaccurately suggesting no cap for childcare eligibility.
The AI failed to identify a well-known income cliff edge, providing misleading eligibility advice.
Key facts
- What
- The GOV.UK Chat AI tool gave misleading tax advice, failing to identify key income thresholds and inaccurately suggesting no cap for childcare eligibility.
- Incident date
- May 15, 2026
- Who
- UK Government
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- Medium
What happened
On May 15, 2026, GOV.UK Chat AI reportedly gave misleading tax advice. According to tax expert Dan Neidle, the bot claimed that the tax-free childcare program has no income limit, which misstates the 100,000 cliff edge; The government also claimed 90 percent accuracy in testing. The tool drew answers from an 80,000-page knowledge base.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
The system failed to accurately retrieve and apply specific tax rules from its 80,000-page knowledge base. It omitted critical conditional constraints, such as the £100,000 income threshold for childcare eligibility.
What it cost
Sources
Cite this entry
https://failureindex.ai/failures/gov-chat-provides-misleading-tax-adviceAI Failure Index. "GOV.UK Chat AI provides misleading tax advice to citizens" (FI-0203). Realm Labs. https://failureindex.ai/failures/gov-chat-provides-misleading-tax-advice (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0203. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
A runtime layer that watches the model's internal state can flag the moment a model commits to a claim it has no support for, and hold or reroute the response before it reaches a user. Realm reads those signals in real time rather than grading the transcript after the fact.