The team asks for extra budget to raise the AI system's accuracy from 95% to 99%. Four percentage points look like a solid gain. But will the company actually feel the difference?
If the remaining mistakes are cheap to fix, polishing the system further may cost more than it ever saves. If a mistake means a missed contractual deadline, a penalty or a lost customer, even 99% may not be enough.
Before signing off on the spend, you need to know not just how often the AI gets it wrong, but what kind of mistakes it makes and what they cost. And before any of that, what the quoted percentage actually means.
One result, three different percentages
Take a hypothetical set of four documents. Each describes ten upcoming events. The system is supposed to find all 40 of them, with the right dates.
On the first document it finds all ten. On the second it misses one. On the third it finds nine real events and adds one that doesn't exist. On the fourth it finds all ten and invents two more.
Three honest numbers can be read off that result:
- 95% of the real events were found: 38 out of 40
- 92.7% of what the system produced is correct: 38 out of 41
- 25% of the documents were processed without a single error: one out of four
None of them is false. Each answers a different question. The trouble starts when a company picks one, calls it "AI accuracy" and never says what it measures.
If a person uses the generated list as a starting point for checking deadlines, the invented events mostly cost the time it takes to throw them out. If the list triggers tasks automatically, the same mistakes can turn into real operating costs. Missing a real deadline also matters less in an internal working list than where a commitment to a customer is on the line.
A correctness score describes how the system behaves. A business assessment says what that behavior means for the company.
There's no single price for an "AI error"
Before the company starts improving the model, it should sort the mistakes by what follows from them. In practice that means settling at least three things.
First, what happens after the error. Is it enough to fix a document label, or does the mistake feed into further actions, customer contact or an important deadline?
Second, whether the error gets caught before it does harm. A mistake picked up by a check may cost a few minutes of work. The same mistake slipping through may mean unwinding a decision made on bad data.
Third, whether reducing one kind of error makes another kind worse. A system can get better at the ordinary cases and, at the same time, stumble more often in the rare, expensive ones. A higher average score is no guarantee of improvement exactly where the company needs it most.
Only with that distinction in hand is it worth talking about money. The consequences can't always be priced precisely, but even a cost range, or a list of errors that are simply unacceptable, beats assuming every mistake costs the same.
What is a jump from 95% to 99% really worth?
Now take a separate, entirely hypothetical example. A company classifies 10,000 documents a month, assigning each one a category. The current system gets 95% of them right, and the proposed upgrade is meant to raise that to 99%, under the same definition of what counts as correct.
At 95%, the company gets 500 wrong labels a month. At 99% it would get 100. The difference is 400 mistakes.
Assume too that the company already checks every label. For this simplified calculation, let's say the check catches every mistake, and that verifying a ready-made label takes less work than classifying the document from scratch. The checking effort is the same at 95% and at 99%. Fixing a mistake once it's found costs 5 PLN.
In that scenario, the monthly cost of corrections drops from 2,500 PLN to 500 PLN. The saving is 2,000 PLN a month.
If reaching and maintaining 99% costs an extra 3,000 PLN a month, it doesn't pay on correction savings alone. If it costs 1,000 PLN, the math looks different. Even then, that's not yet a full assessment of the solution. How to work out the business case for the whole automation is something we covered in a separate article.
In this example, nothing is saved on the checking itself, because every document is still reviewed. We also assumed every error is caught and none causes further losses. In a real company those conditions may not hold.
The point isn't that 99% is worse than 95%. It's that the extra four percentage points may cost more than they're worth in this particular process.
When does this calculation lead to the wrong decision?
Change just one assumption: some of the wrong labels slip past the check unnoticed. Now the saving on corrections is only part of what a better system is worth. You have to add the consequences of the mistakes that were acted on.
The same goes if moving from 95% to 99% genuinely lets the company check less. That's an additional benefit. But it can't be written into the calculation in advance while every document still has to be reviewed.
Averages need care too. A newer model may make fewer mistakes overall but more in a small group of cases with particularly serious consequences. In that situation, the headline improvement alone can lead to a bad purchasing decision.
At 10,000 documents a month, even 99% accuracy means 100 wrong classifications. If just some of them could mean a missed contractual deadline, a delayed order or a lost key customer, a 99% score on its own isn't enough to approve running the process without extra safeguards. You have to check which documents the remaining errors land in and whether they can be caught before any damage is done.
Not every consequence of an error comes down to the cost of fixing it. A mistake that goes unnoticed can mean a late delivery, a contractual penalty, a lost customer or damage to the company's reputation. If losses like that can't be kept in check, a low average cost of corrections doesn't justify full automation.
Instead of improving the model, you can change the way the work is done
When the current system makes mistakes, the best answer isn't always a more expensive model or another few months of tuning.
Sometimes it pays more to handle the typical cases automatically and route the unusual ones to a person. Sometimes it's enough to keep the existing model and tighten the review of its output before anything is done with it. A process like that can deliver a good end result without forcing the AI to resolve every case on its own.
Moving work to people isn't free, though. Someone has to recognize the cases that need a look, check them and decide. People make mistakes too, especially when they trust the system's suggestion without question.
So for a solution that combines AI and human work, measure more than the accuracy of the automatic answers. Measure how many cases the system really handles on its own, how many go to review, and what the whole process delivers.
Four decisions a CEO can make
Before approving further spend on AI quality, ask the team for a side-by-side of the current setup and the proposed one: on the same kind of cases, at a comparable volume, over the same period.
Have them show which errors are supposed to disappear, which will remain, how they're detected, and what it costs both to handle them and to upgrade the system. If one key number is unknown, say the share of mistakes that go undetected, first decide whether finding it out could change the decision.
On that basis there are four reasonable options:
- Keep improving, if the change removes the errors that matter and the value of the improvement justifies the extra cost
- Hold the current level with a check in place, if the remaining mistakes can be caught early and reliably, and further improvement costs more than handling them
- Narrow the scope of automation, if the system handles ordinary cases well but shouldn't decide some of them on its own
- Suspend automatic decisions, if a risk the company can't accept remains despite the safeguards
The highest possible accuracy isn't always the best investment target. Before paying for more percentage points, know exactly which errors they will remove and what that change will do for the company.
And when the test report shows 1,000 runs and zero errors? What a result like that actually proves is the subject of the next part.