AI legal research hallucinations: what studies found and what to check

AI legal research hallucinations: what Stanford measured

Stanford HAI's "AI on Trial" study found the Lexis+ AI and Ask Practical Law AI systems produced incorrect information more than 17% of the time. Its update of May 30, 2024 added Westlaw's AI-Assisted Research, which hallucinated more than 34% of the time.

The study did not cover CaseRead, Paxton, Clearbrief or current versions of any tool, so its rates describe no tool today. Lexis+ AI is now sold as Lexis+ with Protégé.

Read the figures as a measure of those products on those queries in 2024.

General chatbots on legal questions

Stanford HAI's May 2024 report says its previous study of general-purpose chatbots found they hallucinated between 58% and 82% of the time on legal queries. OpenAI says individual ChatGPT plans may use your content to train its models, and you can turn that off under Settings, Data controls. A solo litigator weighing a chatbot should read that term before pasting in client facts.

Why a tool that shows its sources is easier to check

CaseRead says it searches your firm's private documents and public case law together and cites every source. When it lacks a jurisdiction, CaseRead says so and points to Westlaw or Lexis for confirmation.

Shepard's Verify in Lexis+ with Protégé checks citations in AI responses inside the Lexis research and drafting workflow. The link to a source gives you a place to start; it does not replace reading it.

CaseRead lists what its law database covers: statutes and regulations for federal law, all 50 states, DC and Puerto Rico, and case law from the Free Law Project. It lists what it leaves out, including treatises, law reviews, practice guides, jury verdicts and local ordinances.

What a hallucination check can and cannot catch

CaseRead's terms say its AI may produce fabricated information, including invented citations, and that the attorney is solely responsible for verifying every citation. Its About page puts it briefly: "It can make mistakes, so check every citation."

The Hallucination Shield is, in CaseRead's words, an automated screen, not a citator. A "not found" result means the lookup returned no match; it is not proof of invention. Stanford's page also notes that, as of May 2024, more than 25 federal judges had issued standing orders on AI, so read your own court's current orders. AI fake case citations shows what courts have done about the real ones.

Choosing a research tool with this in mind

CaseRead answers from public statutes and Free Law Project case law, links each source, and on Pro and Team flags any citation it cannot ground before you read it as settled. Best AI legal research tools sets it beside Westlaw Advantage and Lexis+ with Protégé, which fit a firm that needs KeyCite or Shepard's inside the research workflow.

Pro and Team at CaseRead start with a 7-day trial, so you can set its cited answers beside your usual source. The CaseRead review lists what each plan includes.

For client files, CaseRead says it runs no model training on your data and routes AI requests to Anthropic and OpenAI through their commercial APIs.