Bobby Kovalsky

For many years, companies have measured the effectiveness of automated systems, such as their chatbot or IVR, by the system’s Containment Rate—the percent of interactions that don’t reach a human agent.

For digital chat programs, optimizing a bot to increase Containment Rate certainly has benefits. If some customers have their problems resolved in fully automated experiences without engaging an agent, then agents’ time will be freed up to assist other customers. Solving more customers’ issues in the bot means companies may require fewer employees to handle chat volume, resulting in cost savings.

What containment rate doesn’t measure

The problem with only measuring Containment Rate is that deflecting a customer doesn’t mean they’ve had their issue resolved. It simply means that a digital agent didn’t get involved in that particular interaction.

In my role as a Customer Experience Strategist at ASAPP, my team and I work with our customers to ensure that their customers are receiving the best possible experience when interacting with the brand. To optimize the customer’s experience, we need to ensure the metrics we are tracking and measuring are answering the key question “Did I resolve my customer’s issue?”

At ASAPP, when judging the effectiveness of a bot and making decisions to improve that effectiveness, we recommend using a metric called Flow Success—the number of conversations where the customer was provided with information necessary to address their issue without the need for a rep to get involved. Using this metric enables companies to understand when their containment is “good containment” and unlocks additional opportunities to optimize their bots towards a great experience.

Why flow success?

It is possible for a chatbot to have a high Containment Rate but a low Flow Success Rate. While this may represent potential cost savings for the company, this is an extremely frustrating experience for the end user.

Some automated flows require customers to take multiple, sometimes unnecessary, steps to find the solution to their problem. Other times, customers may be forced to log in to their account before they can get information when the solution could be provided to them on the phone without logging in. Sometimes customers may choose the wrong path in a flow and give up when they get information that isn’t relevant. These are all examples of “bad containment,” counting towards high Containment but low Flow Success.

In a best case scenario, the customer abandons the experience because they found the answer to their question elsewhere. However, there is a greater risk that the customer gets frustrated with their bot experience and calls instead, forcing involvement from a voice agent, ultimately increasing the cost to resolve that issue. Even worse, a customer may become so annoyed that they become a churn risk for the company. The loss in customer lifetime value can greatly outweigh the cost of having that customer interact with a digital agent.

When shifting the focus from Containment Rate to Flow Success, we are able to help our customers identify and fix areas where this may be happening.

It’s important for CX teams to understand not just whether their automation contained the customer, but whether the customer’s need was actually served.

- Bobby Kovalsky, Sr. Manager, Insights and Strategy, ASAPP

‍

For example, when we analyzed a US cable company’s virtual assistant experience, we found a large gap in Containment and Flow Success for a billing intent. Customers asking to have their bill explained were often contained within the bot but were rarely provided with the information they wanted about their bill. Further analysis revealed that customers were frustrated by the amount of information they needed to provide the bot before the bot would give them their answer. To improve the experience, we recommended the company remove some of these steps, which we ultimately were unnecessary in determining the response the customer needed.

After the company implemented our optimization recommendation, the automated flow not only saw improved Flow Success but it ultimately led to greater Containment. The share of conversations with this intent that were Contained without Flow Success decreased by 21%. Because customers were easily able to access their information in the bot experience, they were less likely to ask to speak to an agent, leading to a 29% increase in Containment Rate.

What are the trade offs?

We’ve seen that organizations that focus on Flow Success rather than strictly Containment are able to create a better customer experience. However, this sometimes means customers will be able to more easily reach a representative. Increasing “good containment” and reducing “bad containment” does not always correlate with an increase in overall Containment.

For example, an internet service provider saw high levels of Containment when customers were asking if they were in an outage. After conducting an analysis to identify areas where customers were not being told whether or not they were in an outage, we found that the existing authentication process was causing customers to abandon the bot. We recommended the company revisit their existing process.

By simplifying the sign-in process, the company ultimately made it easier for customers to reach digital reps and Containment decreased by 3%. However, significantly more customers were informed about the status of their outage, leading to a 17% increase in Flow Success. This organization accepted the tradeoff, allowing more customers to reach digital agents knowing that the large dropoff from the previous sign-in process contributed to increased call volume and therefore higher overall costs.

Although these types of changes may lead to lower Containment, they will ultimately drive higher organizational throughput. By enhancing the digital experience, customers will be more likely to choose digital channels for their future contacts. As the contact mix shifts towards digital, companies unlock additional benefits unique to the channel, such as increasing concurrency, which enable them to handle more conversations with fewer representatives. This leads to larger cost savings than they would have achieved by preventing customers from reaching a digital rep.

What else should be considered?

Measuring Flow Success helps companies analyze and optimize their bot but it is not the only metric that matters. Companies may also want to consider the bot’s contribution to first contact resolution, call prevention, and customer satisfaction.

Creating the best bot experience requires companies to continuously evaluate and optimize performance. Those who focus on delivering the best customer experience in the bot rather than just lowering costs see long term benefits through increased customer satisfaction and higher digital adoption.