During a routine review of complaints closed by the AI, an employee goes back to the customers' original messages and finds that in several cases the system spotted the late delivery but missed the damaged product reported in the same message. It granted compensation and marked the complaint as resolved, even though it hadn't dealt with everything the customer asked for. This is the same case the previous part of this series used to show how to set the limits of automation.
Technical monitoring flagged nothing. The application was running, the integrations were responding, and every one of those tickets had the status "resolved". If nobody had checked the original messages, the company could have gone a long time without knowing its automation was making wrong decisions.
This is one of the biggest challenges in running AI-based systems. Watching uptime and integration errors isn't enough. You have to observe how decisions are made, check regularly that they're right, and act before the same mistake reaches more customers.
With AI, correctness has to be checked after launch too
Conventional software needs monitoring, testing and maintenance as well. AI-based systems add a particular difficulty: the application can carry out every step correctly in technical terms while the model has misread the content or made the wrong call.
Pre-launch tests let you assess the solution on prepared cases. Once it's live, though, customers phrase things in new ways, combine requests unexpectedly and bring situations the earlier tests never covered. Results can also shift after an update to the model, the instructions, the knowledge base or the business rules. Even without any update, a model doesn't always give identical answers to similar inputs.
So passing acceptance tests is not a lasting guarantee of quality. A live system needs regular evaluation of real cases. That doesn't mean checking every message by hand or running the full test suite around the clock. It means planned oversight in proportion to the volume of cases, the kind of decisions being made and what a mistake would cost.
What can you observe in the application itself?
To keep track of how the AI behaves, the right events and records have to be designed into the application from the start. Recording that the model call succeeded is not enough.
For every case it handles, the system can record what type of case the AI identified, what action it proposed, whether it handed the case to a person, what was actually carried out and what status the complaint ended up with. It helps to tie this to the versions of the model, instructions and rules in use, so you can later tell one configuration's behavior from another's. The original message and the source data have to stay available for review by authorized staff, but that doesn't mean copying every piece of personal data into every log.
From these events you can build reports and dashboards showing, for example, the share of cases closed automatically, how often cases are handed to staff, the types of outcome, decisions overturned by staff and tickets reopened by customers. Break the metrics down by comparable case categories and system versions. A single average across all complaints can hide a drop in quality in one less common category.
You can observe the AI's answers and decisions too, not just server status. Some checks run automatically: the application checks a message saying an operation was completed against the actual state of that operation in the source system, detects missing required information, or routes an unusual decision for extra review. You can also add a separate content check, which can be another AI model, to flag answers that may contradict the customer's message or the rules of the process.
An automated evaluator like this can help pick the cases worth checking. It shouldn't be the only proof of correctness, though. A second model can miss the very same problem, or wrongly question a decision that was right. To confirm some errors conclusively, you need independent data or the judgment of someone who knows the process.
None of this necessarily calls for a dedicated AI platform. Depending on scale, well-designed application events, a database report, a quality review queue and notifications may be all you need. What matters more than the name of the tool is the range of information it actually lets you check.
How do you measure quality that doesn't show up in the logs?
Back to the customer's missed request. If the AI never recognized the mention of a damaged product, the system may hold no record of that claim at all. Comparing statuses against the categories the AI itself saved won't uncover anything.
What's needed is a regular review of a sample of real cases, based on the source material. An employee who knows the customer service rules reads the original message, checks the relevant data and compares them with the decision and the reply sent to the customer. They assess whether every request was recognized, whether there were grounds for the decision and whether the case was closed too early. If they start from the summary the AI wrote, they may miss exactly what the model missed.
The sample should include randomly selected cases that were closed automatically. With the right selection and sample size, this lets you estimate how often errors occur in that group, although the result always carries some uncertainty. Higher-risk cases deserve a separate look: messages covering several issues, unusual decisions, complaints about the service, or cases picked out by the automated evaluator. Don't merge the results of that targeted review with the random sample into a single "system error rate", because the cases weren't chosen in the same way.
It also pays to review some of the tickets handed to a person. The AI may be deciding correctly on the cases it keeps while handing staff more and more of the simple cases it used to handle on its own. The quality of automated decisions will look fine, and the benefits of the deployment will quietly shrink. That's why the accuracy of decisions and whether each escalation was justified are assessed separately, against the agreed rules of the process. The goal isn't to cut the number of handoffs at any price.
Record the results by type of mistake, not just as one percentage: a missed request, an unjustified decision, an incorrect message to the customer, a case closed too early. That tells you which part of the automation needs fixing or scaling back. The rate of repeat contacts and the number of complaints about how a case was handled are extra signals, but they don't capture every error, because some customers never follow up.
This review has to happen on a regular cycle, and its scope or frequency should go up after significant changes to the model, the instructions, the data or the process rules. Regression tests before a change is released are still needed. A production review answers a different question, though: how is the system coping with the real cases coming in right now?
When does a monitoring result call for action?
Not every signal means the same thing. If the application sent a customer an unconditional refund confirmation before the payment system had confirmed the refund, you have a specific communication problem. That alone doesn't tell you whether the transfer eventually went through or whether the model was at fault. You need to establish the state of the transaction, correct the information and check for similar cases.
It's different when the share of automatically closed complaints suddenly rises and the number of cases handed to staff falls. That could be a drop in quality. It could just as easily be a change in the kind of cases coming in. A signal like this should trigger a comparison of similar cases and an extra review of decisions, not an automatic verdict that the AI has failed.
So before the automation goes live, agree not only how errors are counted but also what the team will do with the review results. Two things matter: how often errors occur and which way that is trending, tracked separately for the important case types, and how serious the consequences of the mistakes are. If minor errors in replies are creeping up, you can widen the review and look for the cause. A single confirmed case of an important claim being missed, on the other hand, may call for an urgent check of similar cases and a temporary halt to automatic closing in that category. One overall error rate can't support both decisions.
This ties into how often you review. A monthly review may be enough to watch less significant deviations. It can't be the only safeguard in a process where a wrong decision affects many customers every day. There you need earlier signals, more frequent evaluation or an automatic stop on clearly defined actions.
It makes sense, then, to route what the monitoring reports down one of two paths. A confirmed problem with serious consequences alerts a named person and sets off an agreed response, including blocking a specific function if necessary. A deviation that needs interpreting goes to quality control with a deadline for the analysis and someone responsible for it. Urgent alerts that nobody can act on only add noise and can bury a real incident.
What do you do when the AI starts making bad decisions?
Suppose a review of original complaints turns up several cases where the customer's second request was missed. The person running the review compares them with other cases and reports the problem to the process owner, who weighs the consequences and decides whether to restrict automatic closing of tickets.
There's no need to switch the whole application off straight away. If the other functions work properly, the AI can go on sorting messages, retrieving data and drafting replies. Closing cases in the affected category, however, moves to staff. Someone has to take over that queue, set priorities and make sure customers don't get messages saying their complaint has been resolved while it's still open.
In parallel, the technical team checks whether the problem is linked to a change in the model, the instructions, the data or the process rules. A well-designed event log lets you pull out for re-evaluation the cases handled in a given period, under a specific configuration and in the relevant category. But it isn't enough to find the date of the last update and assume that only cases handled since then could be affected. The scope has to be confirmed on real cases.
Fixing the cause and fixing the consequences are two separate jobs. After the fix, new tickets may be handled correctly, but complaints closed earlier still need to be reassessed, and sometimes the customer has to be contacted. The system should get its autonomy back only once the corrected behavior has been verified, there's a plan for resolving the affected cases, and the person responsible for the process has made the decision.
Who makes sure the oversight actually works?
In a live process, responsibility can't end with "the vendor monitors the system". You need to name, separately, who sets the criteria for a correct result, who carries out the independent review, who acts on the findings and who can restrict or restore automated operation.
For the incident described here, a practical split looks like this. The person responsible for quality control confirms the missed requests and identifies the sample cases. The process owner assesses the risk, decides to limit the system's autonomy for a time and orders a re-review of the cases that may have been affected. Service staff put things right for the customers. The technical team or the vendor finds the cause and ships the fix. Only after checking the results does the process owner approve the return to automatic closing of tickets.
These tasks don't have to belong to four different people. They do need specific owners and an agreed way of passing information along. The contract with the vendor should treat these as separate items: infrastructure maintenance, monitoring the quality of AI decisions, access to the necessary data, incident support and responsibility for agreed fixes. Using an external model doesn't automatically shift responsibility for handling complaints onto its provider.
The most important test is a practical one: once a wrong decision is found, can we say which similar cases may have been handled incorrectly, who can stop further automated decisions, and on what basis the system will be allowed back to work?
AI doesn't require checking everything by hand. It does require a planned way of observing real decisions, assessing them independently and correcting course when quality no longer meets what the process demands. A working system isn't just one that's technically available. It's one whose results the company can regularly verify and is prepared to answer for.