A lot of modern AI use is like killing a cockroach with a rocket-propelled grenade.
Or, it’s like a law firm sending every piece of incoming mail to its most senior litigation partner. She opens each one, decides whether it’s a pleading, an invoice, a subpoena, or a client’s holiday card, and drops it in the right tray. Sure, she’s excellent at it. But she’s also the most expensive mail sorter the profession has ever produced. And on the afternoon she’s tired and drops a privileged memo in the wrong tray, nobody catches it, because nobody audits the senior partner’s sorting.
No firm works like this. Law firms are obsessive about leverage: partners do partner work, associates do associate work, and paralegals and software handle the rest. A lot of the business model involves matching the person to the task.
Yet, this is how a lot of firms are deploying AI. Using a large language model for every task is like a senior partner opening every envelope.
AI is not a synonym for LLM
I’m assuming that you’ve already decided that AI is the right kind of tool for your problem. As I’ve argued before, sometimes it isn’t, and you shouldn’t let anyone convince you that you must get on the AI train or be left behind. That’s bullshit. My argument here is narrower. When most people say “AI” today, they mean LLMs, and that conflation may be costing law firms money.
I suspect it also explains a good share of the public backlash against AI and the appetite for building endless data centers. Closer to home, it’s one reason AI spending and AI ROI keep failing to line up. Firms start the conversation with the biggest, most general model on the market instead of with the task they need done. And pretty much every new product pitch pushes in the same direction: bigger models, bigger datasets, more features. That’s backwards.
Most firms also can’t tell whether it’s working. In Thomson Reuters’ 2026 AI in Professional Services Report, organization-wide AI use across professional services nearly doubled in a year, from 22% to 40%. The share of respondents who knew their organization tracked AI’s return on investment stayed roughly flat, at 18%, meaning that although AI adoption is accelerating, measurement is not.
But the lack of measurement isn’t the only problem. A deployment can show real gains and still be a bad deal if a simpler, cheaper system would have delivered the same result. For example, classifying a document, checking a deadline, or pulling a standard field from a form doesn’t call for the machinery you’d want for reconciling conflicting testimony across four hundred pages.
If your AI can write sonnets in Old Norse and you’re using it to decide whether a PDF is an invoice, you’re overprovisioned.
The AI provisioning gap
I call the distance between what a task needs and what a firm deploys the AI provisioning gap. The goal of legal AI deployment should be to minimize this gap, which runs in both directions. Overprovisioning is when you deploy an expensive, general-purpose AI when narrower systems would satisfy the task’s requirements—like the senior partner sorting the mail. Underprovisioning is the opposite mistake, when you use a system too limited to handle the task’s complexity, leading to more errors, delays, or the need for constant human intervention. Underprovisioning is just as economically inefficient. An underpowered tool that makes enough mistakes will create a verification tax, the cost of checking AI output before you can rely on it, until it cancels out any benefits of using AI at all. That’s what I’ve called the Verification Trap.
In my field of computer architecture, the provisioning gap is an old problem. More computing is not better computing. You get the most work per dollar and per watt by matching the hardware to the workload as closely as you can. Google, for example, built its Tensor Processing Unit around one operation, matrix multiplication, because that’s where neural networks spend most of their time. NVIDIA went the same direction when it added dedicated tensor cores to its GPUs. A good deal of my research, including cache memories that can be reconfigured to fit whatever program is running, rests on the same idea.
The catch is flexibility. A chip built for matrix multiplication is a poor choice for nearly everything else, which is the hardware version of underprovisioning. The target is finding the closest match in either direction.
Legal AI deployment deserves the same discipline.
The future of legal AI isn’t bigger models, but better-matched ones.
The legal AI ladder: right-sizing models to the task
The legal AI ladder. Each rung trades capability against computational cost, verification burden, and privacy exposure. The goal is right-provisioning: use the lowest rung that reliably meets the task’s requirements. The rungs can also be combined: for example, an LLM can help develop smaller task-specific models and deterministic rules to handle routine production tasks.
I think of AI options as a ladder of approaches with different capabilities, costs, and tradeoffs. Just like with a regular ladder, you only climb if you have to, and you only climb as high as the task requires, because every rung adds cost and complexity that must earn its keep.
At the bottom rung are rules and lookups, which handle well-defined tasks like deadline calculations, filing checks, and document routing with minimal computing resources and maximum predictability, since they are deterministic models. Predictability does not necessarily mean accuracy—it just means that when the system gets something wrong, you can more easily see why and fix it.
One rung up is classical machine learning, which learns patterns from data and can perform tasks like classification, e-discovery prioritization, and privilege detection with measurable error rates.
Next come small specialized models, including small language models (SLMs), which offer greater flexibility for messier tasks such as clause extraction, client intake processing, transcription, and document summarization, often on modest local hardware.
At the top are general-purpose large language models (LLMs), whose broader capabilities make them useful for complex drafting and summarization, cross-document synthesis, and open-ended analysis, at the highest computational cost and, usually, with the most data leaving the building.
The different rungs are not mutually exclusive. For example, an LLM can help develop a more complete rule set, or label the training data for a specialized classifier that subsequently handles thousands of documents without requiring the LLM again. A more powerful model is only better if its additional capabilities produce enough value to justify its cost.
Right-provisioning goes beyond the choice of model. Computer architects also ask many other provisioning questions that are relevant to legal AI deployment:
| Architectural consideration | Legal AI equivalent |
|---|---|
| Compute requirements | Rules versus classifiers versus SLMs versus LLMs |
| Memory footprint | Model size, context requirements, local hardware |
| Data movement | Sending confidential documents to cloud services versus processing locally |
| Workload specialization | A model trained for contract classification versus a general-purpose LLM |
| Execution efficiency | Cost and latency per successfully completed task |
| Resource utilization | Paying for general-purpose model capabilities that the workflow seldom uses |
The costs beyond computation: verification tax, error consequences, and privacy exposure
The compute bill is only the visible cost of the provisioning gap.
First, there’s the verification tax. Where a system sits on the ladder changes how expensive it is to check its output. When a rule-based system gets something wrong, you can usually trace the error to a specific rule or a missing exception. A classical machine learning classifier can be scored against a labeled dataset, with low-confidence predictions routed to a human for review. LLMs present a different challenge: their fluent, plausible outputs can make errors harder to detect, particularly in tasks involving legal analysis or synthesis. This doesn’t mean simpler systems are necessarily more accurate, or that LLMs always require more verification. It means that verification costs depend on the task and on how easily the model’s errors surface. A model that costs pennies to run but requires substantial human review can get very expensive once you add the verification tax.
Then there’s the expected loss from errors. Misclassifying an administrative document and failing to identify privileged material are both classification errors, but their potential consequences are not comparable. Only one of them can end up in front of a judge. The useful question is how likely a model’s mistakes are to slip through, and what they could cost the firm when they do.
Privacy introduces another architectural consideration. A small, locally deployed classifier may allow a firm to process confidential documents without transmitting them to an external provider, reducing certain exposure risks. That doesn’t automatically make local deployment secure, any more than cloud deployment is inherently unsafe. But where the computation happens changes your exposure.
Put together, the real cost of an AI task looks like this:
\[ \text{Cost per task} = C_{compute} + C_{verify} + (P_{miss} \times L_{error}) \]The first term is what you pay to run the model; the second is what you pay for people or systems to check it; the third is the chance an error gets past them multiplied by what that error costs the firm. Your conversations about AI cost shouldn’t stop at the first term.
The cheapest model to run is not necessarily the cheapest, or safest, model to deploy.
Finding the right rung
Finding the right rung has implications not only for the economics of deployment, but also for environmental sustainability, which I won’t go into here. In general, I recommend the following four steps in deciding what kind of AI to deploy.
| Step | Description |
|---|---|
| 1. Describe the task | Start by describing the task in simple terms: what goes in, what comes out, tests for success, and the consequences of a wrong answer. |
| 2. Start at the bottom rung | Ask whether rules and lookups can do the job. The answer is “mostly” more often than vendors would like you to believe. |
| 3. Climb with criteria | Climb only when the current rung fails a test, such as accuracy on a sample of your own documents or the share of cases it can’t handle. |
| 4. Price the outcome | Compare rungs on cost per task, with verification and error costs included, using the formula above. |
The economics of right-provisioned legal AI
To make the idea of right-provisioning more concrete, consider a small law firm that processes 10,000 documents each month and needs each one classified. It could send every document to a general-purpose LLM through an API, or run a small classifier developed specifically for that task. Assuming both are accurate enough, how much does the rung matter?
Start with compute, since that’s where most cost conversations end. At list prices, sending 10,000 documents of about 3,000 tokens each to a mid-tier model like GPT-5.6 Terra costs around $70 a month as of November 2026 (assuming 100 output tokens, reasoning turned off). A small classifier on a dedicated machine costs about the same once you count the human time to keep it running, and cheaper LLM tiers cost less than either. On compute cost alone, the decision between a cloud-based general-purpose LLM and a bespoke classifier is a rounding error.
Now, suppose a paralegal spends three minutes on each document that needs a human look, at a cost to the firm of $50 per hour. If one system sends just 1% more documents to review than the other, that’s 100 extra documents, five extra hours, and $250 a month—more than the entire compute bill.
Then there can be errors. A single privileged memo that slips into a production can cost more than years of running both systems.
So, it is not sufficient to only ask which model is cheaper to run; you must also ask which model’s mistakes are easier to catch. A classifier gives you a confidence score you can test against your own labeled documents and use to decide what goes to a human, whereas an LLM’s confidence is harder to measure. This is why you should test both on a sample of your own documents before choosing, and not assume either one wins.
Cost calculator
This calculator helps you explore different assumptions and visualize how the cost per task changes when you vary the compute cost, verification cost, error probability, and cost of errors.
What does each document really cost?
Most AI cost comparisons stop at the compute bill. This one adds what you pay people to check the output and what the errors that get past them cost. Both systems start with the same review and error rates, so the defaults don't pick a winner. Change one assumption at a time and watch which bar moves.
- Compute
- Verification
- Errors that slip through
- Compute
- Verification
- Errors that slip through
Method: LLM compute = documents × (tokens × input price + 100 output tokens × output price) / 1,000,000. This assumes reasoning is turned off; reasoning tokens are billed as output and can multiply the output cost. Classifier compute = the monthly running cost you enter. Verification = documents × share sent to review × minutes per review / 60 × reviewer hourly cost. Errors = documents × missed errors per 100,000 / 100,000 × average cost per missed error, an expected monthly cost rather than a bill you'd see every month. In practice, review and missed errors are linked: reviewing more usually means missing less. This model treats them as separate inputs, so change them together.
What's left out: building and training the classifier, integration, and rework on errors that review catches. LLM prices are OpenAI's listed API rates for the GPT-5.6 family (as of November 2026; the Sol rate is a limited-time promotional price); if your firm pays per seat for a legal AI platform, its compute cost is likely higher than this. Every other default is illustrative, and the point is to see which term dominates.
What you should do next
The argument here is not that small models always win, but that capability has a price. That price should be justified by need, not by availability.
My challenge is for you to take a look at the AI workflows your firm currently runs. Pick a high-volume, low-judgment one like intake classification or document routing. Ask which rung it actually needs, then estimate each term of the cost per task formula.
Doing this will enable you to better explain your costs, and more easily trace errors back to a cause. It will also help you prevent unnecessary exposure, like sending out documents that shouldn’t leave the building. You will also have a clearer sense of your ROI, which would already put you ahead of most of the profession.
Firms that don’t perform this exercise will keep doing the equivalent of paying senior-partner rates for mail sorting while the verification tax quietly eats the savings. And sooner or later, one of those unchecked envelopes will be a privileged memo in the wrong tray.