If a program adds up the lines on an invoice, checking it is easy. Enter the amounts, run the function, compare the result with the total you expected. The test passes or it fails.

But what does a passing test mean when a language model is doing a big part of the job?

Say a company wants an LLM to spot urgent customer requests. The model gets ten different messages right. That's a good result for those ten messages. It says nothing about the next ten, written differently. And it doesn't tell you whether the same request, sent again, would land in the same queue.

Ten correct answers prove ten correct answers. They don't tell you how often the system will be wrong, or in what way.

Why testing an LLM is a different job

The classic tests still apply to an application with AI inside it. Calculations, integrations, permissions, data writes and side effects all need checking as before. But once a large language model is part of the application, a new difficulty appears. The same text can be understood in more than one way, and the same answer can be phrased in more than one way. Some of those differences are harmless. Others change the outcome of the whole process. Even with identical input and identical instructions, you can't assume the model will answer identically every time.

Ordinary software can misbehave too, through bugs or because its output depends on something like the current date. What's new is judging the quality of a component whose correct answer doesn't always come in one fixed form.

Which raises the obvious question: how many successful runs are enough? One? Ten? A hundred?

The count on its own settles nothing. Ten right answers to ten nearly identical, easy requests prove nothing about the messy variety a company actually receives. What you need is an agreed definition of "correct" and measurements taken on the real task.

First decide what "correct" means

In our example, the system labels requests as urgent or routine. Before anyone judges its answers, the company has to decide what makes a request urgent.

Is it only when the customer's operations have stopped? What about a safety issue, a deadline that can't be met or a clause in the contract? What about a message that says "urgent" but is really just asking for a document? Without those decisions, two people can look at the same case and disagree. Then there's no way to tell whether the model got it wrong or the rules were never set.

Here, the correct result is the right label under the agreed criteria. The model might write "needs immediate attention" or "urgent request". If both lead to the same correct decision, the difference in wording isn't a mistake.

It's a different story when an identical request is routed to urgent handling once and dropped into the routine queue the next time. Now the business outcome has changed, not the phrasing.

Systems that generate longer text add one more risk. The model may invent a detail that isn't in the documents or the data, like promising the customer a repair date nobody confirmed. That's not the same as a misclassification. Both need controlling, but they're different problems.

What 95% accuracy can hide

Take a simple, hypothetical set of 1,000 customer requests. Fifty genuinely need an urgent response. The other 950 are routine.

The system labels every single one as routine.

It's right 950 times out of 1,000. That's 95% overall accuracy. Respectable, until you remember what the system was for.

It caught none of the 50 urgent requests.

This isn't a real model's result, just arithmetic that exposes the weakness of a single headline number. A system can post a high accuracy score because most cases fall into the easy, common category, and still fail completely at the one job the company bought it for.

That's why a classification like this needs at least two measures.

How many of the genuinely urgent requests did it catch? That's recall, also called sensitivity. If it missed every urgent case, recall for that category is zero.

How many of its alarms were justified? That's precision. A system can flag too many routine requests as urgent and bury the team in false alarms. In our extreme example it raised no alarms at all, so there's nothing to calculate precision from.

These are two different mistakes: a miss, where a case that needed action was overlooked, and a false alarm, where a routine case was escalated. A single accuracy percentage hides the balance between them.

One more thing. A score across a thousand different requests still says nothing about whether the same request gets the same label twice in a row. That's a separate property, and it needs its own check.

Can hallucinations and inconsistency be eliminated?

There's a lot you can do in an LLM-based system: constrain the output format, add rules, check claims against sources, validate results and route unclear cases to a person. Sometimes one plain rule in the application prevents a specific error better than yet another paragraph of instructions to the model.

These measures work. They can cut particular kinds of mistakes dramatically. But reducing risk is not the same as proving that no future answer will ever be wrong. A long streak of correct results is still not that proof.

Variation between answers can be reduced too, through how the model is configured and how the whole process is built around it. What nobody should promise is perfect repeatability just because of a single configuration setting. In most real applications, what matters anyway is that the decision stays consistent, not that every generated sentence is word-for-word identical.

If a vendor tells you their solution "doesn't hallucinate", ask what exactly they mean. Is the output checked against an agreed source? Can the system say it doesn't know? What does it do when the information isn't there? And which errors did they still see in testing?

A newer model isn't automatically better for you

A clear definition of "correct" pays off again when a new model hits the market. It may be faster, cheaper per call or higher on the public leaderboards. None of that means it will label your customers' requests more accurately.

The only comparison that counts is on your tasks, against your quality criteria. After that check you can talk about the new model's advantages. Swapping the model name in a config file is easy. The change may still require adjusting integrations and instructions, and it certainly requires measuring quality again. Price per call alone shouldn't decide the switch.

What to ask before you sign off

You don't have to compute metrics or design test suites yourself. You should, however, expect five things from your team or your vendor:

1. What exactly does the system do? One clear action, such as labeling a request urgent or routine
2. What counts as a correct result? Agreed criteria, with examples of correct, incorrect and borderline answers
3. Which mistakes do we distinguish? Misses, false alarms, missing details and invented details, whichever apply to the task
4. What is the claimed accuracy based on? How many cases were tested and of what kind, how many errors of each type showed up, and what the percentage actually measures
5. Was the stability of the key decision checked? Can the same task, run again, produce a different outcome, even when both answers look convincing?

Without these, nobody in the company can say what the test results actually prove. And that's too thin a basis for signing off on a system.

A test result is not a business decision

For software built on an LLM, "does it work?" is the wrong question. The useful one is: in which tasks does it answer correctly, how often is it wrong, and what kind of wrong? Answering that takes agreed criteria and honest measurements, not one impressive example from a demo.

Even "99.8% of tested tasks correct" doesn't close the subject. It describes a result under specific conditions, not a guarantee for the future. The real question is what happens in the cases where the system does get it wrong, and whether the company can live with the consequences.