Kohls v Ellison: Expert AI declaration excluded for fake citations
In Kohls v Ellison, a Stanford professor submitted an AI‑assisted expert declaration that contained fake citations; the court excluded the declaration and criticized the use of AI in the filing, underscoring the need to verify AI outputs in legal submissions.
AI hallucinations in expert filings can destroy credibility in court.
Key facts
- What
- In Kohls v Ellison, a Stanford professor submitted an AI‑assisted expert declaration that contained fake citations; the court excluded the declaration and criticized the use of AI in the filing, underscoring the need to verify AI outputs in legal submissions.
- Incident date
- Jan 10, 2025
- Who
- Minnesota Attorney General's Office
- Failure mode
- Hallucination
- AI surface
- Chatbot
- Severity
- High
What happened
A Stanford professor submitted a declaration drafted with GPT-4o to support the Minnesota AG in Kohls v Ellison. The declaration allegedly contained fake citations and was excluded by the District of Minnesota, with the court noting the irony of using AI to critique AI. The decision described the declaration as unreliable and the credibility of the expert as shattered.
What broke inside the model
- 01 · TriggerA user asks for a fact, a citation, or a figure.
- 02 · Model stepThe model writes a fluent, confident answer.
- 03 · Control gapNothing ties the claim back to a real source.
- 04 · FailureA fabricated fact ships as if it were verified.
- 05 · ConsequenceThe false claim reaches a customer, a court, or the public.
Confidence holds, and even spikes, as the claim detaches from any source.
The expert drafted the declaration with GPT-4o, produced hallucinated/misattributed citations, and signed it under penalty of perjury, prompting the court to exclude the testimony as unreliable.
What it cost
Sources
- Court FilingORDER granting in part and denying in part 29 Plaintiffs' Motion to Exclude Expert Testimonylaw.justia.com
- PressThe Irony of Using Generative AI in a Case About the Dangers of Generative AIbracewell.com
Cite this entry
https://failureindex.ai/failures/kohls-ellison-expert-declaration-excluded-fakeAI Failure Index. "Kohls v Ellison: Expert AI declaration excluded for fake citations" (FI-0275). Realm Labs. https://failureindex.ai/failures/kohls-ellison-expert-declaration-excluded-fake (indexed Jun 5, 2026).Data fields CC-BY 4.0, prose citation permitted. Incident ID FI-0275. Full dataset at /data.
Note from Realm Labs, the Index steward
How Realm would have caught this
- Prism
- OmniGuard
- AI Detection & Response (AIDR)
A runtime layer that watches the model's internal state can flag the moment a model commits to a claim it has no support for, and hold or reroute the response before it reaches a user. Realm reads those signals in real time rather than grading the transcript after the fact.