When AI Invents Legal References
Artificial intelligence continues to infiltrate sectors where rigor is non-negotiable. Latest case in point: the prestigious American law firm Sullivan & Cromwell had to issue a public apology after a document filed in court contained citations… entirely fabricated by an AI tool. A phenomenon that specialists call “hallucinations,” this unfortunate tendency of language models to manufacture information with bewildering confidence.
Andrew Dietderich, a partner at the firm, acknowledged that while the company does have internal procedures to prevent such slip-ups, they simply weren’t followed in this particular case. In other words: the safeguards existed, but no one applied them. A classic case of human error, compounded by perhaps excessive faith in the machine.
What a “Hallucination” Means in Simple Terms
To understand what happened, a brief technical detour is necessary. Large language models (LLMs), like those powering tools such as ChatGPT and its competitors, generate text by statistically predicting the most likely words in a given context. The problem: they don’t really “know” what they’re saying. When asked to cite case law, they can easily invent one that seems perfectly plausible — case number, party names, date — but never actually existed.
In a judicial context, submitting a false reference to a court is not trivial. It can range from embarrassing corrections to disciplinary sanctions for the lawyers involved. Sullivan & Cromwell, one of Wall Street’s most prestigious firms, got off relatively lightly this time with an apology and a promise to strengthen internal processes. Phew.
The Legal World’s Response: Train First
Faced with these pitfalls, the legal sector is starting to organize. Thousands of miles away from Sullivan & Cromwell’s New York office, Mississippi College School of Law decided to take the bull by the horns. Recently, artificial intelligence training became mandatory for all first-year students.
The idea is straightforward: rather than ignoring AI or banning it outright, it’s better to teach future lawyers to use it responsibly — and especially to identify its limitations. Understanding that a tool can “hallucinate” citations is exactly the kind of critical knowledge that can prevent an embarrassing incident before a judge.
This initiative is part of a broader movement: American courts themselves are beginning to legislate on AI use in judicial proceedings, with some now requiring lawyers to explicitly declare whether they used an AI tool to prepare their documents.
A Tension Revealing a Universal Challenge
What’s happening in the legal world is actually a mirror of what many sectors face with generative AI. On one hand, the technology offers considerable productivity gains: document drafting, precedent research, contract analysis. On the other, it introduces new error vectors, sometimes insidious precisely because the output appears convincing.
The parallel with the crypto universe is no accident. In a space where players rush into new technologies — often before regulatory frameworks are in place — Sullivan & Cromwell’s mishap reminds us that adopting a tool doesn’t exempt you from understanding how it works. No technology, however promising, replaces human judgment and verification processes.
Perspective
The Sullivan & Cromwell case and Mississippi College School of Law’s initiative illustrate two complementary responses to the same challenge: how to integrate AI without falling prey to its flaws. Upfront training and downstream control procedures emerge as the two essential pillars of responsible adoption.
As AI enters sensitive domains like justice, medicine, or finance, the question is no longer whether to use it, but how to use it. And clearly, even the world’s best law firms still have a few lessons to learn — which, in a way, reassures all of us about our own relationship with technology.
