Lawyering is thinking. Delegating it to a machine hollows out the craft.
For thirty years, lawyers have worried about what they might overlook in the age of digital search. Now the tools themselves are fabricating what was never there. When Sullivan & Cromwell filed motions in 2026 citing cases that do not exist, it revealed something deeper than a compliance failure: a profession at risk of outsourcing the very thinking that defines it. The machine does not reason — it performs the appearance of reasoning — and in law, that distinction is everything.
- A partner at one of America's most prestigious law firms had to apologize to a federal judge after filing a motion filled with invented citations and misquoted statutes — and that firm advises OpenAI on responsible AI deployment.
- More than 1,100 court filings have now been flagged by judges for AI-introduced errors, signaling that this is not a series of isolated mistakes but a systemic pattern taking hold across the legal profession.
- The danger is not that AI is obviously wrong — it is that AI sounds fluent and confident while fabricating facts, flattening complex legal distinctions, and hallucinating case law that has never existed.
- Lawyers are most tempted to delegate drafting, the very stage where precision and voice matter most, risking the reduction of legal writing to polished, authoritative-sounding nonsense.
- The profession is navigating toward a more disciplined model: AI as a pressure-testing sparring partner in advocacy preparation, not as a substitute for the slow, foundational thinking that frames a case from the start.
Three decades ago, legal scholar Patti Ogden warned that the ease of searching Westlaw could breed dangerous overconfidence — that lawyers might miss what the tool made them feel they had already found. The worry then was about human oversight. The worry now is different: the machines themselves are inventing.
In May 2026, a partner at Sullivan & Cromwell sent an apology to a New York bankruptcy judge. A motion filed ten days earlier contained citations to cases that did not exist and statutes that had been misquoted. The firm, notably, advises OpenAI on responsible AI deployment. It was not alone in its failure — a public database now documents more than 1,100 court filings where judges have identified AI-introduced errors.
To understand why this matters, consider what lawyering actually is. It moves through four stages: framing the legal questions from the facts, researching the governing law, drafting the filing, and arguing before the court. An error in the first stage cascades through all the others.
Framing is where AI is most dangerous. It requires a lawyer to sit with the facts slowly, letting the legal questions emerge from the material. AI skips this entirely — producing instant, confident, and often wrong lists of issues, sometimes built around facts that were never in the record. Research carries its own hazard: AI systems routinely cite cases that appear real and are entirely fabricated, and even when a citation is accurate, the analysis tends to flatten the four or five distinct sub-issues that careful research would surface.
Drafting is where the temptation is strongest and the stakes highest. A draft is what the judge reads. Used carelessly, AI becomes the author rather than the editor, and a profession already prone to boilerplate risks losing the capacity for precision entirely. The one stage where these tools genuinely help is oral argument preparation — AI makes a tireless, probing sparring partner once the case has been properly built by human hands.
Sullivan & Cromwell had policies. The firm understood the risks. The error happened anyway. The lesson is not that AI must be avoided, but that the thinking cannot be delegated. Lawyering is judgment. The craft and the cognition are not separable — and no tool that merely performs the appearance of reasoning can substitute for either.
Three decades ago, a legal researcher named Patti Ogden raised an alarm about the databases that were beginning to reshape how lawyers worked. The ease of searching Westlaw, she warned, could breed a dangerous overconfidence—a lawyer might miss a crucial case simply because the tool made finding things feel so effortless. The worry was about what humans might overlook. Now the problem has inverted. The machines themselves are inventing.
In May 2026, a partner at Sullivan & Cromwell—one of the country's most established law firms—sent an apology to a New York bankruptcy judge. A motion the firm had filed ten days earlier contained citations that did not exist and statutes that had been misquoted. The firm, notably, advises OpenAI on how to deploy artificial intelligence responsibly. This was not an isolated incident. A public database now documents more than 1,100 court filings where judges have identified errors introduced by AI systems. The pattern is becoming impossible to ignore.
To understand why these errors matter so much, it helps to trace what a lawyer actually does. Litigation moves through four distinct stages, each building on the last. First comes the work of understanding the facts and framing the legal questions at stake. Second is research—finding the cases and statutes that govern the dispute. Third is drafting the motion or brief that will be filed with the court. Fourth is the oral argument itself, the moment when a lawyer stands before a judge and makes the case. An error in the first stage cascades through all the others like a line of dominoes.
The first stage is where machines are most dangerous and least useful. Framing an issue well requires a lawyer to sit with the facts, to read them carefully, to let the legal questions emerge from the material itself. This is slow work. It cannot be rushed. An AI system, by contrast, is seductive precisely because it is fluent. Ask it to identify the issues in a dispute and it will produce a list instantly, sounding confident and complete. The problem is that the list will often be wrong. The model reaches for plausibility, inventing facts that were never mentioned in the brief and constructing legal issues around these fabrications.
Research is where Ogden's three-decade-old warning finds its modern echo. A lawyer asks an AI system for case law on a particular point, and the system obliges—often with citations that appear entirely real and are entirely false. Even when a cited case actually exists, the AI's treatment of it tends to be shallow. Most legal questions, examined closely, break apart into four or five distinct sub-issues, each governed by a different line of authority. The machine flattens these distinctions. It also lacks access to the paid databases where most case law actually lives, the archives that practicing lawyers rely on every day. The best use of AI at this stage is as a rough starting point, a preliminary map of the terrain—nothing more.
Drafting is where the temptation becomes strongest. A draft is what the court will read. What the judge will see. Used carefully, an AI can help pressure-test a draft for gaps and internal contradictions. Used carelessly, it becomes the author of the draft itself. A profession already criticized for formulaic, boilerplate prose will then lose something it can never recover: the ability to write with precision and voice.
Advocacy—the fourth stage, the oral argument—is where these systems are actually useful. Once a case has been properly prepared, an AI makes a patient sparring partner. It will play the role of a hostile judge, probing the weak points in an argument, testing the logic. It will not tire. It will not let you off easy.
Sullivan & Cromwell had policies in place. The firm knew the risks. The error happened anyway. This is the hard truth: lawyering is thinking. It is reading. It is judgment. Delegating any of the four stages to a machine—outsourcing the work to something that sounds intelligent but cannot actually reason—hollows out the craft itself. The thinking is not separate from the lawyering. It is the lawyering.
Citações Notáveis
The model reaches for plausibility, inventing facts that were never in the brief and building issues around them.— Analysis of how AI systems fail at the first stage of litigation
Lawyering is, above all, thinking. Delegating any of the four stages to a machine hollows out the reading and thinking on which the craft depends.— Core argument presented in the piece
A Conversa do Hearth Outra perspectiva sobre a história
Why does it matter that Sullivan & Cromwell made this mistake? They're one firm. They apologized.
Because they advise OpenAI. They have resources, expertise, policies. If they can't control the technology, who can? It signals that even careful deployment fails.
But couldn't a lawyer just check the citations? Verify the cases before filing?
Yes. And that's the point. If you have to verify everything the machine produces, you've added work, not saved it. You're reading twice. The temptation is to trust it.
The piece says AI is useful for advocacy—for oral argument. Why is that different?
Because by then the thinking is done. The case is framed, the research is finished, the draft is written. The lawyer knows the terrain. The AI is just a sparring partner, testing arguments that the lawyer has already built.
So the danger is delegation at the wrong stage?
Exactly. Delegation at the beginning, where judgment is most needed. That's where the thinking lives.
What would responsible use look like?
A lawyer reads the facts carefully, frames the issues themselves, uses AI to map the research landscape, writes the draft themselves, then uses AI to find holes. The machine is a tool for pressure-testing, not for thinking.
And if lawyers don't do that?
Then we get more fabricated citations, more invented cases, more judges catching errors that should never have been filed. The profession loses credibility. The courts lose time.