A model produced fluent, plausible text. A lawyer read fluency as accuracy. The court treated the filing as the lawyer’s. Every litigator has now heard that story — and has usually heard it as a 2023 problem, solved by better tools. The count says otherwise, the purpose-built tools still err, and the same three failures run quietly through document review, where nobody checks a citation.
The most famous failures of legal AI were not exotic. A model produced fluent, plausible text. A lawyer treated fluency as accuracy. The court treated the filing as the lawyer’s. That pattern — hallucination, plus a weak prompt, plus automation bias — is now an ethics and sanctions problem rather than a technology curiosity.
It is almost always told as a brief-writing story about a consumer chatbot in 2023, which is exactly why it gets under-applied. Neither half of that framing has held up: the incident count has risen every year since, and preregistered testing of the purpose-built legal research tools sold as the fix found them erring on a meaningful share of queries too. The same three failures operate in discovery, where the output is not a citation anyone can check in thirty seconds but a responsiveness call across 40,000 documents, a privilege designation, or a log entry. A wrong tag does not announce itself the week it happens. This piece is about how each failure actually works and what stops it. It assumes you have already decided to use these tools; if the question you have is what record a court will want afterward, that is a different question and we have written about what “defensible” actually means separately.
The short version: a language model predicts the next token, so a real citation and a fabricated one cost it exactly the same effort — and purpose-built legal tools reduce that rate without getting it near zero. The prompt, not the platform, is your coding manual. And the human failure — accepting output because it is fluent, or because it matches the case theory — is what turns a model error into a filing. Ground the task in the document, separate generation from authority, and design the review so that a reviewer cannot rubber-stamp without noticing.
A hallucination is confident output that is not grounded in a real source: a case that does not exist, a quote the document does not contain, a date the email never stated. IBM lists it among the core limitations of large language models, and Judicio explains why it is structural rather than incidental. The model samples the next probable token. A well-formed reporter citation — volume, reporter abbreviation, page, court, year — is a highly probable pattern. Existence is not required for the pattern to look right.
The consequence worth sitting with is that hallucination is not a defect a vendor patches out in the next release. It is the same mechanism that makes the model useful, running without a source to check itself against. Everything below follows from that: the fixes are all about constraining what the model is allowed to answer from, and about who checks the answer before it leaves the building.
Two features of how these systems are built make law an unusually exposed domain. The first is the long tail. Consistent patterns — spelling, grammar, the shape of a citation — are learned reliably; arbitrary low-frequency facts cannot be inferred from patterns at all. A specific docket number, a jurisdiction-specific holding, the date on one email in one custodian’s mailbox: these are exactly the facts a general-purpose model is least able to reproduce, and legal work runs almost entirely on them. The second is an incentive problem that OpenAI’s researchers described in 2025. The benchmarks used to evaluate and reward models score correctness, not calibrated uncertainty. A confident guess scores the same as knowledge and better than an honest “I don’t know,” so the training and evaluation pipeline has been quietly rewarding confident fabrication over abstention. That is a property of the pipeline, not of any one vendor’s product.
Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023) is the cautionary holding everyone cites because it is concrete. Counsel opposing a motion to dismiss submitted a brief citing cases that did not exist: fictional airlines, invented docket numbers, quotations attributed to judges who never wrote them. When opposing counsel could not locate the decisions and the court ordered copies produced, the attorneys did not check a database. They went back to ChatGPT, which assured them the cases were genuine and available on Westlaw and LexisNexis. They were not.
Judge P. Kevin Castel described portions of the submission as “gibberish,” found subjective bad faith under Rule 11, and imposed a $5,000 penalty along with letters to the client and to the real judges whose names had been attached to fake opinions. The opinion is equally important for what it does not say. Using a reliable AI tool is not inherently improper. Abandoning the gatekeeping role is — and the sanction followed the doubling down at least as much as the original error. Asking the model whether the model was right is the purest form of the failure this whole piece is about.
The comfortable reading of Mata is that it was 2023, the tool was a consumer chatbot, and the profession has since learned. The docket does not support that reading.
Within months, Michael Cohen disclosed that he had passed AI-generated citations produced with Google Bard to his own attorney, who filed them in support of a request to end his supervised release early. The court declined to sanction Cohen, but the episode moved the risk outside the population of litigators who had been warned. A client, a paralegal, a self-represented party, an expert’s assistant — anyone feeding model output into the pipeline can introduce the same defect, and the signature at the bottom of the filing still belongs to a lawyer.
Since then the count has gone up rather than down. A project tracking court dockets had documented more than 1,300 proceedings across over 100 countries involving AI-fabricated content submitted to courts and tribunals as of April 2026, several hundred of them involving licensed attorneys, spanning family, bankruptcy, immigration, appellate, and criminal matters. Sanctions have escalated on roughly the same curve: $5,000 in Mata in 2023, about $11,000 in a 2024 case, individual penalties above $55,000 by early 2026, alongside a growing number of referrals to state bar disciplinary authorities. Courts in the United Kingdom, Singapore, Australia, Canada, and Argentina have reached materially similar conclusions without any apparent coordination between them.
The institutional response has moved in step. A substantial number of federal and state courts now operate standing orders requiring counsel to certify that AI-generated content in a filing was independently verified by a human being — which is worth checking before you negotiate an ESI protocol that is silent on the subject.
The other comfortable reading is that this is a consumer-chatbot problem, solved by moving to a legal research product built on retrieval-augmented generation — where the system searches a curated database of real authority before answering rather than answering from memory. RAG helps, substantially. It does not close the gap, because a system can retrieve the wrong authority, correctly retrieve the right one and then misstate what it says, or attach a fabricated quotation to a citation that is entirely real.
That is not a theoretical objection. Stanford’s RegLab ran a preregistered empirical study of leading legal AI research products — Lexis+ AI, Westlaw’s AI-Assisted Research, and Thomson Reuters’ Ask Practical Law AI — against vendor claims of having eliminated hallucination through retrieval. The tools produced incorrect or unsupported answers on a meaningful share of tested queries, reported in the range of roughly one in six to one in three depending on the product and on how an error was counted. Substantially better than an unguarded chatbot. Nowhere near the marketing.
We are a legal AI vendor writing that paragraph, so the implication should be stated plainly rather than left as an inference about competitors. “Hallucination-free” is not a claim anyone in this category has earned, ours included, and a buyer should read it as a marketing adjective rather than a specification. The useful questions are narrower and harder to answer: what is the measured error rate, on which task, against what ground truth, and who did the measuring. That is the reason we publish our evaluation methodology and the benchmark numbers themselves rather than an adjective — and the reason the mitigations at the end of this piece assume a residual error rate rather than pretending one away.
In document review, hallucination looks different and is considerably harder to catch. The model rarely invents a Federal Reporter citation inside a responsiveness rationale, because it is not being asked about the law. It invents a meeting that “must have” happened, attributes a statement to the wrong custodian on a forty-message thread, drops the word “draft” from a description, or reads a boilerplate confidentiality footer as the sender’s own assertion of privilege. These are the same long-tail facts described above — who said what, when, in which version — and a corpus of one company’s email is nothing but long tail.
None of that produces a show-cause order the week it occurs. It distorts issue coding, privilege calls, and deposition outlines, and it surfaces — if it surfaces — months later, when a witness contradicts the chronology you built. The brief-writing version of this failure is partly self-limiting, because opposing counsel checks the citations within days. The discovery version has no equivalent adversary until production.
Scroll table horizontally →
| Failure | In a brief | In document review |
|---|---|---|
| Fabricated source | A case, quote, or pin cite that does not exist | A meeting, attendee, or date the document never mentions |
| Dropped qualifier | Dicta described as a holding | “Draft,” “proposed,” or “hypothetical” lost from a summary |
| Misattribution | Right proposition, wrong court or year | Right statement, wrong custodian on a long thread |
| Who catches it | Opposing counsel, within days | Possibly nobody, until a deposition or a clawback |
| What it costs | Rule 11 exposure and a published opinion | A waiver fight, a bad production call, or a theory built on a document that says something else |
In a traditional review the coding manual is the control surface. It defines responsive, defines each issue tag, gives examples of close calls, and is written by someone who has read the complaint. In an LLM workflow the prompt is the coding manual — plus the role, the constraints, and the output schema. Treating it as configuration rather than as a protocol is the second failure in the sequence, and it is the one most within your control.
Vague prompts produce vague and sometimes invented structure. “Find the smoking guns” is an invitation to narrate drama, and the model will oblige. “Apply the following definition of responsive, quote the sentence that supports your call, and answer Unknown if the document does not say” is a narrower instruction that produces a checkable result.
A workable matter prompt usually carries five things:
Everlaw and DISCO both describe prompt iteration on samples as the generative-AI analogue of training-set design in TAR. That analogy holds only if someone with case knowledge writes and versions the prompt. Delegating “prompt engineering” to a vendor who has not read the complaint is the same mistake as letting a contract reviewer invent the issue codes — and it is a mistake that shows up in the measurement, which is why we built a formal evaluation framework for our own review output rather than eyeballing samples.
Prompts also raise work-product and disclosure questions that are genuinely unsettled. Paul Hastings’ discussion of Schulte notes that review protocols have generally been treated as work product, and that how much of an AI prompt a court might later expect to see is an open question. A prompt drafted by a senior lawyer encoding the theory of the case looks a great deal like work product; mechanically, it also functions exactly like a search term, and parties exchange those routinely. Nobody should want to litigate that by surprise in a Rule 37 motion after the review is finished. Label prompts, keep every version and the reason for each revision, raise AI use at the Rule 26(f) conference or during ESI protocol negotiations, and do not volunteer the entire instruction set in a declaration unless you intend to live with the waiver argument that follows.
Automation bias is the tendency to accept machine output because it is machine output. Confirmation bias is the tendency to accept it because it matches what you already believe about the case. Generative systems aggravate both at once: the prose is polished, the rationale reads like an associate wrote it, and the first summary you read becomes the story of the case before a contrary document has had a chance to be found.
ABA Formal Opinion 512, issued July 29, 2024, does not use the social-science vocabulary, but it describes the duty that counters both. Model Rule 1.1 requires competence, including a reasonable understanding of the benefits and risks of the tools in use. The opinion warns that generative AI can produce inaccurate, incomplete, or biased results, that these systems do not “understand” meaning the way lawyers do, and that they can generate plausible falsehoods. Using such a tool without an appropriate degree of independent verification can itself violate Rule 1.1. Thomson Reuters’ summary of the ethics rules ties the same duty to Comment 8’s technology-competence language, to Rules 5.1 and 5.3 on supervising lawyers and nonlawyer assistance, and to Rule 3.3 candor toward the tribunal.
Courts have been blunter about the same duty. A good-faith belief that the tool’s output was accurate is not, by itself, a defense to filing fabricated material — as one court put it, a lawyer cannot rely on “an empty head and a pure heart.” The standard is objectively reasonable verification, and the fact that a machine produced the error does not move it. Opinion 512 also reaches the invoice: Rule 1.5 constrains what a lawyer may bill a client for when adopting the technology, which is worth thinking about before an efficiency gain is quietly retained as margin.
Academic commentary has started to name the psychology expressly. Anna Conley’s work on the duty of competence describes competent generative-AI use as informed decision-making plus verification, and warns against automation complacency — the slow loss of the ability to do the task you have handed to the model. In discovery that is not a faculty-lounge concern. A reviewer who only confirms AI tags will not notice that the prompt never defined “business confidential,” or that the model is treating every email with an in-house lawyer on the cc line as privileged, which is one of the most reliable ways to draw a challenge to a privilege log.
Confirmation bias has a discovery-specific form worth naming on its own. Using the model to “find support” for a theory and then treating the returned excerpts as a complete record is a search, not a review. A prompt that asks only for documents showing scienter will not surface the exculpatory thread, and it will not tell you that it did not. Sampling the null set, or deliberately running a contrary prompt, is the old TAR discipline that still applies without modification.
Mitigation here is operational, not rhetorical. None of the following is novel; all of it is what gets skipped when a deadline compresses.
Two practical consequences follow, and both are staffing decisions rather than software ones. The first is that whoever writes and versions the prompt is doing protocol work and should be as senior as the person who would otherwise have written the coding manual. The second is that the QC role changes character. Spot-checking a junior reviewer means looking for errors in a process that is visibly imperfect; spot-checking a model means looking for errors in a process that is right often enough to lull you. That calls for a different sample, drawn deliberately from the null set and the low-confidence tail rather than from the middle.
Model choice matters too, though not in the way procurement usually assumes. Our nine-model responsiveness benchmark held the written responsiveness definition identical across every run precisely so the prompt could not confound the comparison, and found that model selection rather than model tier determines accuracy on this task — in the earlier eleven-model study, the most expensive model tested scored lower than one costing 54 times less. The same pattern holds in our work on model selection for privilege review. But a better model dropped into a workflow with no validation step is only a faster way to reach the same place.
None of this requires rejecting generative tools, and neither Mata nor Opinion 512 suggests it. Both assume lawyers will use them. What they require is the stance you already take toward a first-year associate: useful, fast, unsupervised at your peril. That is the premise our own platform is built on — the model narrows the population and drafts the first pass, an attorney makes every call that leaves the building, and the audit trail is generated as the review runs rather than reconstructed under deadline afterward. The same discipline applies to AI-drafted privilege log entries, where a hallucinated description is a Rule 26(g) problem attached to a document you have already withheld.
The tell, in the end, is emotional. If an output feels like a relief — because it confirms the case theory, or because it finishes the brief — slow down. Fluency is not a citation. A citation is not a holding. A holding is not a production decision. Those gaps are where professional judgment still earns its fee.
What is an AI hallucination in a legal context?
Confident output that is not grounded in a real source: a case that does not exist, a quote a document does not contain, a date an email never stated. It is a property of how the model works rather than a defect in a particular product. A language model samples the next probable token, and a well-formed reporter citation is a highly probable pattern — the citation does not have to exist for the pattern to look right.
Can hallucinations be eliminated by using a better model?
They can be reduced substantially and they cannot be eliminated by model choice alone, because the mechanism that produces them is the same one that makes the model useful. What reduces them in practice is architecture and process: retrieve the actual document and require the model to quote from it rather than answer from memory, never let the model be the source of a citation or a rule, and verify output in proportion to what it will be used for.
Do purpose-built legal AI tools still hallucinate, or does retrieval solve it?
They still do, at rates worth planning around. Retrieval-augmented generation — having the system search a curated database of real authority before answering, rather than answering from memory — reduces the problem substantially, and it does not close it: a system can retrieve the wrong authority, correctly retrieve one and then misstate what it says, or attach a fabricated quotation to a real citation. Stanford’s RegLab tested leading legal research products in a preregistered study and found incorrect or unsupported answers on a meaningful share of queries despite vendor claims of elimination. Treat “hallucination-free” as marketing language and ask instead for a measured error rate, on a stated task, against stated ground truth.
How often are lawyers actually getting caught?
Often enough that it is no longer a novelty defense. A project tracking court dockets had documented more than 1,300 proceedings across over 100 countries involving AI-fabricated content as of April 2026, several hundred of them involving licensed attorneys, across family, bankruptcy, immigration, appellate, and criminal matters. Sanctions have escalated alongside the count — $5,000 in Mata in 2023, roughly $11,000 in a 2024 case, individual penalties above $55,000 by early 2026 — with a growing number of referrals to state bar disciplinary authorities and materially similar rulings in the UK, Singapore, Australia, Canada, and Argentina.
Are generative AI prompts discoverable, or are they work product?
Unsettled. Review protocols have generally been treated as work product, and a prompt drafted by a senior lawyer encoding the theory of the case looks a great deal like one — but mechanically a prompt also functions like a search term, and parties routinely exchange those. The practical answer is not to litigate it by surprise: raise AI use at the Rule 26(f) conference or in ESI protocol negotiations, keep versioned prompts, and do not volunteer the full instruction set in a declaration unless you intend to live with the waiver argument.
Does using AI for document review violate the ABA Model Rules?
No. ABA Formal Opinion 512 assumes lawyers will use generative AI. What it requires is competence under Rule 1.1 — a reasonable understanding of the benefits and risks of the tool — and an appropriate degree of independent verification before relying on the output. Using a tool without that verification is what can violate Rule 1.1. Confidentiality under Rule 1.6, supervision under Rules 5.1 and 5.3, and candor under Rule 3.3 apply alongside it.
How do you catch hallucinations in document review, where nobody checks a citation?
By making the source visible and by measuring disagreement rather than trusting a tag. Require the model to cite the Bates number and quote the sentence supporting each call, so a reviewer is checking a claim against text rather than accepting a label. Then validate before scaling: run the prompt on a stratified sample, have attorneys code the same sample blind, measure where the model and the humans disagree, revise the prompt, and re-measure on a fresh sample. Generative review does not improve mid-run the way continuous active learning did, so the validation has to be front-loaded.
What is automation bias, and how do you design a review against it?
Automation bias is the tendency to accept machine output because it came from a machine; its companion, confirmation bias, is accepting it because it matches your theory of the case. Design against both structurally rather than by exhortation: show reviewers the source passage instead of only the tag, require a click-through on low-confidence and high-severity codes, rotate QC so the prompt author is not the sole validator, sample the null set as well as the hits, and track override rates. An override rate near zero is more likely to be evidence that nobody is reading than evidence that the model is perfect.
This article is general information about legal technology and discovery practice, not legal advice for any particular matter, and describing a case here is not a representation about how any court would rule on your facts. ABA Formal Opinion 512 is guidance rather than binding authority, and state, federal and jurisdiction-specific rules and standing orders on generative AI continue to change — check your own court’s. The $5,000 figure is the monetary penalty imposed in the Mata sanctions order and is not a general measure of exposure; the later sanction figures and the count of more than 1,300 proceedings come from third-party docket-tracking projects and press reporting as of April 2026 rather than from an official register, and any such count is a floor rather than a total, since it captures only what was detected and reported. The error-rate range for legal research tools is as reported for the Stanford RegLab study and varies by product and by how an error is counted; read the study before relying on a specific number. The 1% override rate is offered as a diagnostic worth investigating, not as a validated threshold — the right number for a given review depends on the population and the tags.