The test report for an AI system shows 1,000 runs and zero errors. The team recommends launching it for customer service. Is that enough to say yes?

We don't even know yet whether those thousand runs included complaints, vague questions or situations that should have been handed to a human agent. Maybe every single test was a simple, repetitive question.

Before going live, check not just the test score but what the tests actually prove.

Testing starts before the AI is switched on

For a customer service system, the natural test material is past conversations, messages and tickets. They show what customers really ask about, how they phrase their requests and which situations give the team trouble.

But it isn't enough to pull the conversation history and check whether the model answers the way an agent did. The agent's original reply may itself have been wrong, based on an outdated price list or out of step with today's company policy. The data may also contain dozens of near-identical questions that artificially inflate the score.

So preparing test material takes two kinds of work. The people who own the process decide what the right outcome is: when to answer, when to decline and when to hand over to a person. The technical team turns those decisions into repeatable tests, runs them and reports the failures. Contested cases have to be settled in advance, or flagged as needing a separate judgment.

The same applies to an AI that recognizes products on photos of receipts. Someone has to check which products are really there and decide what to do with an unreadable line. Without a trustworthy reference answer, a wrong AI result can pass as correct.

When real customer conversations are used, there also has to be a legal basis for using them, a limit on personal data and proper safeguards. Deleting the surname alone isn't always enough.

This isn't admin work around the tests. Its quality decides whether the test measures the right thing at all.

Typical cases are no substitute for the hardest ones

A test set should answer two different questions.

How will the system cope with everyday work? That calls for the most representative cross-section you can get: simple questions, complaints, requests to change details or to speak to an agent, in the proportions real support actually sees.

Does the system fail exactly where it must not? That takes a separate set of deliberately hard cases. Say a customer gives contradictory information, demands something the bot can't do, or needs to reach a person urgently. It's also worth checking that the AI doesn't escalate nearly everything just to be safe.

In contract analysis, an ordinary sample may contain almost none of the rare, hard-to-spot obligations or deadlines. Build separate cases for them instead of hoping they turn up by chance.

Don't fold the results of these two sets into a single percentage without thinking. The first describes typical work. The second is where you go looking for weaknesses on purpose. You need both.

A thousand successful runs don't prove 99.99% accuracy

It also matters what "1,000 tests" means. A thousand repetitions of the same conversation check how stable the answer is for one specific case. A thousand different cases give a broader picture of what the AI may run into. But even a thousand different cases can be badly chosen.

There's no universal number of tests after which a system counts as ready. How much checking is needed depends on what the AI is supposed to do, how serious its mistakes can be and which actions it will be allowed to take on its own.

Suppose we tested 1,000 independent, representative cases on an unchanged version of the system, with the definition of an error agreed beforehand. We found no mistakes at all. The observed score is 100%, but that doesn't mean the system will be flawless on the next cases. All it tells you is which error rates are still plausible given that result and those assumptions. By a simple statistical rule of thumb, a thousand runs with no failures still leave room for an error rate of around 0.3%. So a thousand successful runs can't be turned into a claim of 99.99% accuracy. What hides behind a single percentage is the subject of our article on AI accuracy.

And what if only 30 of those 1,000 conversations needed to be handed to an agent? Then there are just 30 relevant cases for judging missed escalations, not 1,000. A big number at the top of the report can hide very few tests of the behavior that matters most.

The final check has to be kept apart from the fixes

While building the system, the team runs tests over and over, analyzes the failures and adjusts the model's instructions, its configuration or the integrations. That's how it should work. The problem starts when, after many rounds of fixes, the team declares success on nothing but the same well-worn cases.

You need a separate acceptance set that wasn't used to tune this version of the solution. Only then can you tell whether the improvement holds for cases the system wasn't prepared for. It also helps to have someone from outside the team that built the application take part in the acceptance.

Keep the hard cases discovered earlier as a set of their own. After every significant change, run them again to make sure fixing one error didn't bring back another.

Just as important is deciding before the test what result lets the project move forward and which mistakes block the launch. Changing those rules after seeing the report strips the test of any meaning as an acceptance criterion.

Test the whole customer journey, not just the model's answer

The model alone may correctly conclude that the customer should talk to an agent. But if the application routes the conversation to the wrong team, or drops the contact history on the way, the customer still won't be served properly.

Likewise, an answer can be perfectly phrased and still rest on an outdated price list or data pulled from the wrong account. So the tests should also cover fetching information, calling tools, the actions taken and the handover to a person.

Check too what happens when a data source is down or a write fails. Does the system report the problem and hand over to a person, or does it try to answer without the information it needs? Does retrying a request perform the same action twice?

A correct escalation is sometimes the right test result. There's no point penalizing the system for not answering on its own when that's exactly what it should have done. At the same time, check that it isn't escalating cases it could have resolved safely.

Runs that involve actions such as sending a message, issuing a refund or changing customer details should first happen in a controlled environment, with no chance of accidental effects on real accounts.

A new model has to pass the same exam

After launch, a cheaper or faster language model may come along. Swapping the model in the application is often a small change, but its answers, its data format and the way it uses tools can all differ.

So a new model should go through the same comparable tests: the typical cases, the hard ones and the errors fixed earlier. Then check the whole process, integrations and handovers included. If the change brings new capabilities or new risks, extend the test set.

The same goes for changes to the model's instructions and to important integrations. Only once quality is confirmed is it worth comparing the handling time and the real running cost of the new variant.

Five questions before approving a pilot

The test report doesn't have to be a lengthy technical document. It should, however, make it possible to answer five questions:

1. Where did the test cases come from? Which ones reflect everyday work, and which were chosen to probe the hard and critical situations?
2. Who approved the correct results? Did the business agree on the rules for answering, declining and escalating, and settle the ambiguous cases?
3. How strong are the results? How many different cases, how many repetitions, how many tests covered the most important errors, and was there a separate acceptance test?
4. What didn't work? Which errors remain, were the integrations tested, and does the system hand cases to agents correctly?
5. Did we meet the acceptance criteria set before the tests? Which conditions were met, which errors block the launch, and what scope of pilot can we allow? What safeguards will be in place, and what will stop the pilot?

The answers should lead to a concrete decision: go back to fixes and test again, launch a limited pilot under supervision, or hold the launch. The company might, for instance, let the AI report order statuses but not issue refunds on its own, if tests of that function showed significant errors. Clearing one percentage threshold is not an automatic green light for full deployment. Why a percentage alone isn't enough is the subject of the previous part.

A thousand successful runs may be a good result. What matters more is which situations those runs covered, what was left outside the tests, and what the company is basing its launch decision on. Only a report like that helps manage the risk of deployment instead of hiding it behind an impressive number.