The check the language model skips
The most dangerous AI output is the plausible one: cleanly formatted and still wrong. Why a bigger model does not fix that, and why structure does.
The obvious worry when a law firm uses AI is that the model will get things wrong. But the obvious error rarely does the damage. A visibly nonsensical result gets noticed straight away. What gets dangerous is the result that looks thorough, is cleanly formatted and is still wrong in a few places.
A report that looked like the answer
The freely available book “Building AI-Native Professional Services Firms” by Daniel Katz, Michael Bommarito and Jillian Bommarito tells the building of an AI-native firm as a novel. One scene sticks.
The firm is reviewing a pharmaceutical company’s compliance. Four production sites, each with its own approval, fourteen products that have moved between sites over time. The AI checks every rule against the documentation and produces a report: 180 pages, 47 findings, each with a severity rating and a recommendation. It looks like solid work.
It is not. The AI treats the company as a single entity. It reports violations at a site that is not approved for that product class at all, so against rules that do not apply there. Pure noise. The error in the other direction weighs more: it gives a site a green light for a product that was moved there recently and whose transfer documentation is incomplete. Eleven errors of that kind, three of them serious. It was only caught because the client side had a former regulatory inspector who knew the site logic.
Not a calculation error but a skipped question
The model did not miscalculate. It applied every individual rule to the documentation correctly. What it left out was the question that comes first: does this rule apply to this entity at all?
In legal terms that is a skipped applicability check. Before applying a provision, you establish whether it applies. The AI went straight to subsumption, matching the facts against the rule. It saw documents and rules, but not which rule governs which case. That intermediate step, self-evident to a human with domain knowledge, was missing.
Where the same error lurks in a law firm
Scope of application is rarely clear cut in firm work. A deadline shifts depending on the public holiday in a given federal state. What applies to one corporate form does not apply to the next. And a money laundering due diligence obligation bites in one matter and not in another. In all these cases the dangerous error is not the wrong calculation but the right calculation on the wrong facts.
That is the same observation as behind “calculate, don’t guess”, one level up. There, a deterministic piece of code (one that computes the same way for the same input) does the arithmetic rather than the language model, because “usually right” is the wrong category for deadlines and amounts, the same reason AI adoption belongs in the non-billable work. With scope of application it is not about the arithmetic but about the question before it: which rule governs here? Before a tool applies a provision, it has to establish and show which facts it applies to.
The countermeasure is structure
In the book the fix was not a better model but an intermediate step. Before the actual comparison it resolves the entities: which site, which approval, which product, which transfer. Only then does the system compare, and each finding applies only to the area it was raised for. On top of that came a second review, one that checks not the individual findings but the review scope itself, done by someone with the right domain knowledge.
For the tools I build, that is the same principle as with arithmetic. A proposal that turns up as a finished result gets waved through. A proposal that shows which rule it applied to which facts can be checked. Make the scope of application visible and sign-off becomes a decision again.
Responsibility under professional conduct rules lies with the admitted lawyers regardless. A tool does not take it off their hands, but it can supply the basis on which it can sensibly be exercised.
What I take from it
The most dangerous output is the one that is plausible and wrong at once. A bigger model lowers the error rate but not the error type. It delivers the same confidently wrong output, just less often, and rare errors are harder to spot than frequent ones. The answer is not a question of the model but of the construction: expose the scope of application before the result and make it checkable in thirty seconds.
The content of this blog was created partly with the support of AI.