Bobby Kovalsky

AI agents promise faster service, lower costs, and seamless customer experiences. But are they actually delivering on those promises – or just creating the illusion of efficiency? Many businesses adopt AI solutions expecting major savings, only to discover later that misaligned use cases, vendor hype, and misleading metrics obscure the real picture. 

We want to cut through the noise to show you how to determine whether your AI agent is delivering a solid return on your investment.

When AI agent efficiency is merely an illusion

The promise of AI agents is huge – great customer service at scale with a lower cost to serve. But the savings are not automatic. Simply launching an AI agent that serves customers doesn’t guarantee increased efficiency, a better customer experience, or lower costs for your contact center. 

So beware of these common pitfalls. 

Misaligned use cases

One of the most critical decisions points in a successful AI agent launch is identifying the right use cases. Funnel the wrong customer conversations to your AI agent and you’ll end up with frequent escalations and frustrated customers. A reliable solution vendor should be able to guide you through a proven data-driven process to identify use cases that have high automation potential and will make a big impact in your contact center.

Vanity metrics

As the saying goes, there are three kinds of lies – lies, damned lies, and statistics. It’s not that statistics are inherently dishonest. But the wrong statistics can lead you astray by focusing your attention on the wrong things, so choose carefully which ones you rely on. Metrics like the number of interactions your AI agent handles might sound impressive, but if resolution rate is low, it’s not a meaningful measure of the solution’s impact.

Overpromising vendors

Every vendor extolls the virtues of its solution. But not every vendor delivers a product that lives up to the hype. Look for cold, hard facts when you’re researching solutions. A vendor that can back up their marketing spin with real-world results is worth serious consideration. Look for a solution provider who’s as focused on creating value for your business as you are.

The three metrics that matter most

There are a number of metrics you could – and should – track to measure the performance of an AI agent. But when you’re trying to zero in on whether the solution is saving you money, there are three key metrics that matter most. Taken together, these three measures provide the basis for determining the return on your AI investment.

Resolution rate

It’s not enough for your AI agent to handle and contain customer interactions. To save you money, it has to resolve customer issues. Otherwise, one of two things happens – customers call or chat again, this time determined to reach a human, or they decide your brand is not worth the trouble. Either way, an AI agent that fails to resolve customer issues actually costs you more than it’s worth.

Change in labor hours required

At the heart of an effective AI agent strategy is one overarching objective – to shift interaction volume away from human agents without degrading the customer experience. If your AI agent is simply taking over the load from your IVR and traditional bots, it’s not delivering the savings you’re looking for. To lower your costs, it must reduce the labor hours required to handle the interactions.

Escalation cost

No AI agent today can handle every interaction that comes its way. It will sometimes need to hand off the conversation to a human, or if it can, ask a human for information or guidance so it can continue serving the customer on its own. Every time your AI agent involves a human, the cost for that interaction goes up. You’ll need to know how much those handoffs are costing you.

Quick diagnostic: Are you getting the return you hoped for?

Rigorous ROI models that account for every factor can be complex and time-consuming to run. But you can get a good indication of whether your AI agent is delivering savings with a simple formula. To get started, you only need to know two things:

1. The number of interactions the AI agent kept out of your human agents’ queues
2. The time it would have taken human agents to handle those interactions if your AI agent hadn’t contained and resolved them

Here’s how the data looked for a major US airline using ASAPP’s GenerativeAgent to help passengers rebook flights.

From here, you can calculate the cost savings. If you know the per-interaction or per-minute cost of a human agent in your contact center, you can calculate the savings directly. If you don’t, you can count the minutes saved and determine how many FTEs would be required to handle that volume. 

If your solution is able to consult with a human agent for advice, guidance, and approvals, and then continue resolving the customer’s issue on its own, you’ll need to account for the cost of the FTEs needed for those consults, as well.

This is only a rough calculation, but it’s a good starting point for determining whether your AI agent is actually saving you money. You’ll also want to consider the value of these other possible benefits:

• Faster resolutions: In the case of the airline noted above, GenerativeAgent was able to rebook flights much faster than human agents. An interaction that previously ate up 35 minutes of the customer’s time was now resolved in only 8 minutes. When customers get what they need quickly, they’re less likely to open a second interaction in a different channel to see which one goes faster. That keeps queues short, which helps prevent burnout for your human agents.
• Improved customer satisfaction: Less time on hold, faster resolutions, and more consistent service add up to a better customer experience. That drives customer satisfaction higher.
• Smaller volume fluctuations for human agents: Automating issues with unpredictable volume spikes smooths out fluctuations, which reduces the need to overstaff in preparation for these scenarios.

If the math doesn’t add up…

If you run these simple calculations and don’t see significant savings, you’ll need to dig deeper into what’s actually happening. But don’t be too quick to conclude the solution is never going to deliver savings. You might just need to refine and reconsider some things in your deployment to optimize your agent’s performance and realize more value. Here are a few adjustments to keep in mind.

Is your AI agent handling the right use cases?

Because the goal is to shift volume away from human agents, you want your AI agent to handle as many interactions as possible. But to keep both efficiency and customer satisfaction high, it should only receive interactions it’s capable of resolving. That makes use case definition a foundational step in achieving savings. If your AI agent is falling short, take a hard look at how you’ve defined your use cases and determine whether they should be redefined.

Is your AI agent handling sufficient volume to deliver savings?

With initial deployments, some businesses choose to limit the volume of interactions sent to their AI agent until they’re confident in its performance. It’s a cautious choice, but it can reduce the impact the solution delivers. If it’s performing well, consider throttling up the volume. If you started with a low-volume use case, you’ll want to identify additional use cases that your AI agent can handle so you can expand both its role and its impact in your contact center. Just keep in mind that before you decide to scale up, you need to have a solid understanding of what your AI agent is doing well and where it needs to be fine-tuned. If you scale up a solution that has not yet been optimized, you scale up its deficiencies at the same time. 

Do you need to improve your knowledge assets?

AI agents might seem to know a lot, but they’re not omniscient. They rely on your knowledge articles and other sources of truth. If information is missing from those sources, your AI agent cannot fill in the gaps on its own – and you wouldn’t want it to try. If your knowledge articles contain conflicting or poorly organized information, the AI agent can easily draw the wrong conclusions. So, you’ll want to consider where you need to supplement or improve your knowledge base content.

Don’t just automate. Realize actual savings.

Deploying an AI agent is only the first step. Realizing the value it’s promised depends on continued optimization. Don't just rely on surface-level stats or vendor promises. Instead, focus on the outcomes that actually move the needle: resolution rate, volume shift, and escalation cost. These metrics help you see beyond the illusion of activity to the true financial impact on your contact center.

Ultimately, it’s a question of whether your AI agent is doing the work that saves your business time and money while keeping customers satisfied. Measure what matters, refine continuously, and hold your solution accountable for real results. That’s how you turn automation into actual savings.

For many years, companies have measured the effectiveness of automated systems, such as their chatbot or IVR, by the system’s Containment Rate—the percent of interactions that don’t reach a human agent.

For digital chat programs, optimizing a bot to increase Containment Rate certainly has benefits. If some customers have their problems resolved in fully automated experiences without engaging an agent, then agents’ time will be freed up to assist other customers. Solving more customers’ issues in the bot means companies may require fewer employees to handle chat volume, resulting in cost savings.

What containment rate doesn’t measure

The problem with only measuring Containment Rate is that deflecting a customer doesn’t mean they’ve had their issue resolved. It simply means that a digital agent didn’t get involved in that particular interaction.

In my role as a Customer Experience Strategist at ASAPP, my team and I work with our customers to ensure that their customers are receiving the best possible experience when interacting with the brand. To optimize the customer’s experience, we need to ensure the metrics we are tracking and measuring are answering the key question “Did I resolve my customer’s issue?”

At ASAPP, when judging the effectiveness of a bot and making decisions to improve that effectiveness, we recommend using a metric called Flow Success—the number of conversations where the customer was provided with information necessary to address their issue without the need for a rep to get involved. Using this metric enables companies to understand when their containment is “good containment” and unlocks additional opportunities to optimize their bots towards a great experience.

Why flow success?

It is possible for a chatbot to have a high Containment Rate but a low Flow Success Rate. While this may represent potential cost savings for the company, this is an extremely frustrating experience for the end user.

Some automated flows require customers to take multiple, sometimes unnecessary, steps to find the solution to their problem. Other times, customers may be forced to log in to their account before they can get information when the solution could be provided to them on the phone without logging in. Sometimes customers may choose the wrong path in a flow and give up when they get information that isn’t relevant. These are all examples of “bad containment,” counting towards high Containment but low Flow Success.

In a best case scenario, the customer abandons the experience because they found the answer to their question elsewhere. However, there is a greater risk that the customer gets frustrated with their bot experience and calls instead, forcing involvement from a voice agent, ultimately increasing the cost to resolve that issue. Even worse, a customer may become so annoyed that they become a churn risk for the company. The loss in customer lifetime value can greatly outweigh the cost of having that customer interact with a digital agent.

When shifting the focus from Containment Rate to Flow Success, we are able to help our customers identify and fix areas where this may be happening.

It’s important for CX teams to understand not just whether their automation contained the customer, but whether the customer’s need was actually served.

- Bobby Kovalsky, Sr. Manager, Insights and Strategy, ASAPP

‍

For example, when we analyzed a US cable company’s virtual assistant experience, we found a large gap in Containment and Flow Success for a billing intent. Customers asking to have their bill explained were often contained within the bot but were rarely provided with the information they wanted about their bill. Further analysis revealed that customers were frustrated by the amount of information they needed to provide the bot before the bot would give them their answer. To improve the experience, we recommended the company remove some of these steps, which we ultimately were unnecessary in determining the response the customer needed.

After the company implemented our optimization recommendation, the automated flow not only saw improved Flow Success but it ultimately led to greater Containment. The share of conversations with this intent that were Contained without Flow Success decreased by 21%. Because customers were easily able to access their information in the bot experience, they were less likely to ask to speak to an agent, leading to a 29% increase in Containment Rate.

What are the trade offs?

We’ve seen that organizations that focus on Flow Success rather than strictly Containment are able to create a better customer experience. However, this sometimes means customers will be able to more easily reach a representative. Increasing “good containment” and reducing “bad containment” does not always correlate with an increase in overall Containment.

For example, an internet service provider saw high levels of Containment when customers were asking if they were in an outage. After conducting an analysis to identify areas where customers were not being told whether or not they were in an outage, we found that the existing authentication process was causing customers to abandon the bot. We recommended the company revisit their existing process.

By simplifying the sign-in process, the company ultimately made it easier for customers to reach digital reps and Containment decreased by 3%. However, significantly more customers were informed about the status of their outage, leading to a 17% increase in Flow Success. This organization accepted the tradeoff, allowing more customers to reach digital agents knowing that the large dropoff from the previous sign-in process contributed to increased call volume and therefore higher overall costs.

Although these types of changes may lead to lower Containment, they will ultimately drive higher organizational throughput. By enhancing the digital experience, customers will be more likely to choose digital channels for their future contacts. As the contact mix shifts towards digital, companies unlock additional benefits unique to the channel, such as increasing concurrency, which enable them to handle more conversations with fewer representatives. This leads to larger cost savings than they would have achieved by preventing customers from reaching a digital rep.

What else should be considered?

Measuring Flow Success helps companies analyze and optimize their bot but it is not the only metric that matters. Companies may also want to consider the bot’s contribution to first contact resolution, call prevention, and customer satisfaction.

Creating the best bot experience requires companies to continuously evaluate and optimize performance. Those who focus on delivering the best customer experience in the bot rather than just lowering costs see long term benefits through increased customer satisfaction and higher digital adoption.