ITAT Bengaluru withdraws tax order citing fake AI judgments
The ITAT Bengaluru withdrew a tax order involving Buckeye Trust after discovering it relied on fake legal precedents generated by AI. The incident highlights the risk of using generative AI for legal research without rigorous verification.
The tribunal incorporated fabricated AI-generated case laws into a formal ruling without performing due diligence.
Key facts
- What
- The ITAT Bengaluru withdrew a tax order involving Buckeye Trust after discovering it relied on fake legal precedents generated by AI.
- Incident date
- Dec 1, 2024
- Who
- Income Tax Appellate Tribunal (ITAT) Bengaluru
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- Medium
What happened
The Bengaluru bench of the ITAT released a tax order that cited four non-existent precedents (three from the Supreme Court and one from the Madras High Court). The order was withdrawn within about a week after it became clear the precedents did not exist in legal records. It is alleged that a generative AI tool was used to find favorable verdicts and these results were incorporated without due verification.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
A generative AI tool hallucinated plausible-sounding but entirely fictitious legal citations and case names. This failure was exacerbated by a lack of human verification as the tribunal bench accepted these hallucinations as legal precedent.
What it cost
Sources
- PressDid AI hallucination play mischief with a tax tribunal order?livemint.com
- PressITAT Retracts Order After AI-Generated Fake Judgements in Buckeye Trust Tax Casebhaskarenglish.in
Cite this entry
https://failureindex.ai/failures/itat-bengaluru-withdraws-tax-order-citingAI Failure Index. "ITAT Bengaluru withdraws tax order citing fake AI judgments" (FI-0313). Realm Labs. https://failureindex.ai/failures/itat-bengaluru-withdraws-tax-order-citing (indexed Jun 8, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0313. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
A runtime layer that watches the model's internal state can flag the moment a model commits to a claim it has no support for, and hold or reroute the response before it reaches a user. Realm reads those signals in real time rather than grading the transcript after the fact.