Blog

Imagine you’re booking airline tickets through a conversational AI assistant, and after purchasing tickets, you ask for help in finding an in-home pet sitter during your trip. The conversational AI misinterprets what you mean, and instead shares details on how to board your flight with pets. This has an obvious reason: the AI has never encountered this particular task, and was unable to map it to a procedure. Thus, your request to find an in-home pet sitter was out of the distribution of what the assistant was trained to handle. Alternatively, suppose you had asked about upgrading your flight, but the system confuses your request as wanting to update your flight to a different date. In this case, the AI assistant is capable of managing flights but was unable to complete the request due to a dialogue breakdown. In both cases, we arrive at the same result: a failed conversation.

Both out of distribution requests and dialogue breakdowns described above are considered out-of-scope (OOS) situations since they represent cases that your assistant is unable to handle. To avoid customer frustration, detecting OOS scenarios becomes an essential skill of today’s conversational AI and dialogue systems. While the ideal conversational AI agent would be able to help find an in-home pet sitter as requested and manage all the complex nuances of natural language, this is simply not possible given that training data is finite and consumer queries are not. So knowing when the user is asking something in-scope vs out-of-scope can help refine conversational AI systems into better performing in their core tasks.

It can be hard to provide training data for, or even enumerate, the potentially limitless number of out-of-scope queries a dialogue system may face. However, new ASAPP research presented at the conference on Empirical Methods in Natural Language Processing (EMNLP) offers a novel way to address this limited-data problem.

Out-of-Scope Detection with Data Augmentation

We introduce GOLD (Generating Out-of-scope Labels with Data augmentation), as a new technique that augments existing data to train better out-of-scope detectors operating in low-data regimes. The key insight is that rather than training on in-scope data alone, our proposed method operates on out-of-scope data as well. Furthermore, we discover that common NLP techniques for augmenting in-scope data, such as paraphrasing, do not provide the same benefit when working with out-of-scope data.

GOLD works by starting with a small seed set of known out-of-scope examples. This small amount (only 1% of the training data) is typically used by prior methods for tuning thresholds and other hyperparameters. Instead, GOLD uses this seed set of OOS examples to find semantically similar utterances from an auxiliary dataset, which yields a large set of matches. Next, we create candidate examples by replacing utterances in the known out-of-scope dialogues with the sentences found in extracted matches. Lastly, we filter down candidates to only those which are most likely to be out-of-scope. These pseudo-labeled examples created through data augmentation are then used to train the OOS detector.

The results? State-of-the-art performance across three task-oriented dialogue datasets on multiple metrics. These datasets were created by post-processing existing dialogue corpora spanning multiple domains with multi-turn interactions. Notably, the out-of-scope instances were designed as a natural progression of the conversation, rather than generated through synthetic noise or negative sampling.

Why this matters

Data augmentation is a popular method to improve model performance in low-resource settings, especially in real life settings where annotating more examples can quickly become cost-prohibitive. With just a small seed of out-of-scope examples, GOLD achieved a 10X improvement in training out-of-scope detectors compared to using the seed data alone. Previous methods relied on using tremendous amounts of labeled out-of-scope data that is unrealistic to obtain in real-world settings or relied on in-scope data alone which doesn’t provide sufficient signal for detecting OOS items.

‍

With just a small seed of out-of-scope examples, GOLD achieved a 10X improvement in training out-of-scope detectors compared to using the seed data alone.

Derek Chen

GOLD supports robustness and prevents overfitting by relying on other methods during the filtering process. As other out-of-scope detection methods improve over time, GOLD can take advantage of those gains and improve as well.

At ASAPP, we are exploring similar methods in our products to both reduce out-of-scope issues in our conversational systems, as well as improve overall systems when operating in limited data regimes. If you’re a researcher conducting work to detect more granular levels of errors, or more sophisticated methods of data efficiency, we’d love to chat! Give us a tweet at @ASAPP.

Read our paper on GOLD

At ASAPP, we build AI models to increase the efficiency and effectiveness of customer service agents by recommending the next action to perform during live chat. A natural starting point for creating such models is to learn to predict when the agent would perform the action based on the data we have collected in production. The data we collect are usually of the form: timestamp & event. For example:

14:28:03, AGENT ASSIGNED TO AN ISSUE
14:28:44, AGENT SENDS A MESSAGE
14:29:18, AGENT CLICKS AN AUTOMATED RESPONSE
14:31:52, AGENT ENABLES AUTOMATIC TIMEOUT TO TAKE OVER

For example, our AutoSuggest model learns the likelihood that an agent will send a certain message based on the context of the conversation and other features. Our AutoCompose service uses this model to surface the most likely next message as a suggestion to the agent, reducing their response time, lower cognitive load, and encouraging them to use preferred phrases. The actual next message sent by the agent has proven to be an effective target for AutoCompose, resulting in high usage, reduced average handle time, and positive feedback from agents.

However, sometimes what agents actually do, isn’t necessarily what they should do. And what agents actually do in production is the raw data that’s logged. If we are not careful, this is the data that will be used to train AI models, which will reinforce suboptimal agent behaviors through recommendations and automation.

This was the situation with our Automatic Timeout feature and our Flexible Concurrency feature. Automatic Timeout is an automation feature that agents can opt in to when the customer has likely left the chat. The feature will send timeout messages on behalf of the agent so that the agent can focus their effort elsewhere. The feature was a huge success in increasing agent efficiency and extremely popular with agents.

‍

We discovered that agents would manually time out customers rather than use the feature to avoid receiving additional assignments.

Chris Fox

‍

To improve usage of Automatic Timeout, ASAPP developed a recommendation model to recommend the Automatic Timeout feature to agents. The most natural starting point seemed to be predicting when agents were likely to use the feature, based on the usage data we had collected in production. But, there was a wrinkle.

Soon after Automatic Timeout went live, our AI-driven Flexible Concurrency feature was launched. This feature learns and predicts agent busyness. When it predicts the agent is likely to not be busy, the agent’s concurrency can be increased (flexed) without overwhelming the agent. One of the biggest predictors of an agent’s busyness is whether Automatic Timeout has been enabled. Agents began to notice that there was a correlation between using Automatic Timeout and increased concurrency. Because companies typically tie agent performance to agent handle time (rather than their throughput), agents are not incentivized to take on additional issues. As a result, usage of the Automatic Timeout feature decreased. Agents would manually time out customers rather than use the feature to avoid receiving additional assignments.

Because some agents were not using Automatic Timeout to avoid additional issue assignments, many timestamps where a recommendation would be relevant were incorrectly labeled as times not to recommend.

As an alternative to leveraging agents’ past usage of Automatic Timeout as the prediction target, we explored instead labeling each timestamp based on whether there would be no further customer actions after that point in the chat. This approach had the advantage of not being affected by some agents’ preference to manually time out the customer. It captured all cases where the customer became idle during the chat. Moreover, the model achieved high accuracy on this prediction task.

However, upon further testing, we discovered that this prediction target was in fact not as good a choice as it first appeared. The model was recommending Automatic Timeout very frequently during the end of normal chats, in which the customer issue had been resolved and the agent was closing out the conversation. The predictions were highly confident in these sections of the conversation.

Meanwhile, in cases where the customer had gone idle while the agent was waiting for them to respond, the model often predicted no recommendation or had low confidence. Looking further into the data, the reason was clear: normal chats are far more common than chats in which the customer leaves the agent waiting. As a result, the model focused on detecting normal chat endings, and worse, our evaluation metric was largely reflecting model performance in those irrelevant situations.

This is an example of a common issue in the application of AI models: the usefulness of a model depends on choosing the prediction target carefully. A poorly selected target can result both in a model that is ineffective for its intended application and an evaluation metric that obscures this fact from the model developer.

We considered further restricting the target to require, not only that the customer was inactive, but also that the conversation concludes by being timed out. However, it can be helpful for the agent to use Automatic Timeout to free them up temporarily to focus on other work, even when the customer comes back before the sequence completes in a few minutes.

In the end, we designed a more complex target that better identifies the times in the chat when it would be useful to enable Automatic Timeout, based on additional situational data and an auxiliary model. Specifically, the customer needs to have been idle for a certain period, and the next event in the chat is either an Automatic Timeout message, a timeout-like message sent manually by the agent, or the agent timing out the customer (which closes the chat and prioritizes the customer in the queue if they chat back in again). An auxiliary model is used to identify timeout-like messages. This work was primarily driven by our amazing intern Sara Price.

Data used to label Automatic Timeout recommendation model training data

As you can see from the table above, labeling training data for the Automatic Timeout recommendation model based on what agents should do entails a lot more modeling effort than simply replying on what agents have been doing (using the feature or not). Fortunately, with the ASAPP AI Native® approach, the additional models needed to determine the type of language the agent is using are already available and can be easily consumed by the Automatic Timeout recommendation model.

With the final version of the prediction target, we achieved a better alignment between the training data, and hence the model’s behavior, and the usefulness of our recommendations. And the evaluation metric became a better indicator of how useful our model would be in practice. In some cases, simply predicting agent actions is sufficient to build helpful AI recommendations, but in other cases, as with Automation Timeout, we have found it can pay dividends to think carefully about how to engineer the training data to guide agents toward more optimal workflows.

People dread contacting customer service. It means they have a problem or a need they can’t solve on their own. Experience has shown them it’s rarely quick and easy to resolve issues. Phone calls often come with long hold times, transfers between agents, and explaining the problem again and again.

Many people prefer messaging to phone calls for all their communications—but are met with disappointment when they try to message with their favorite consumer brands.
It’s not that most companies don’t offer messaging or chat in some form. It’s just that trying to resolve an issue by chat is often even worse than calling, so it has not reduced the volume of calls in the way companies hoped.

Why are these systems failing you?

You’ve got pipes.
You need something more.

The problem is that most of these systems were built on chat technology developed in the 1990s—and haven’t evolved much since then. They provide a basic communications channel for conversations between consumers and agents. And, they enable agents to engage with more than one customer at a time.

BUT this legacy technology doesn’t do anything to empower those agents to provide great service for anyone, let alone more than one at a time.

Meanwhile consumer expectations have evolved significantly over the past 30 years. They want companies to value their relationship and demonstrate that with readily available personalized and competent service in the channels they prefer to use. If a company misses, today’s consumers don’t hesitate to call them out in social media. And, with low barriers to switching they don’t stay with brands that don’t get the message.

The oldtimers may say they help agents –
but check the results.

Most chat solutions claim to use AI to help agents solve customer issues. But in reality, they only use simple AI to diagnose intent at the beginning of an interaction, and that’s where it ends. Little or no ‘intelligence’ is applied to the rest of the conversation, so agents do what they’ve always done: Follow a series of rules-based suggestions, rely on their own skills and knowledge (often after only minimal training), hunt for answers in their toolset, consult with other agents, and transfer the contact to someone else when they can’t get resolution.

It’s not surprising that a bit of AI layered on outdated technology is not delivering meaningful results.

Macario Namie

It’s time to REALLY help your agents.

Customer service is long overdue for real innovation. Legacy chat has its place in history. Now it’s time to harness groundbreaking advancements in AI—and realize the full potential of this technology to transform customer experience performance.

What’s possible when you reimagine customer experience with AI as foundational technology? Machine learning can be infused in every aspect of digital care interactions in an integrated system that will:

• Provide AI-driven predictive suggestions based on continuous learning, to guide agents on what to say and what to do to resolve issues quickly.
• Dynamically integrate automation into the workflow to handle routine tasks before, during, and after agent interaction, reducing agent workload.
• Enable rich personalization by providing agents with the full context of the consumer’s interactions before this engagement.
• Use a multivariate model to manage concurrency for agents—considering factors like agent experience, complexity of issues already engaged, customer history and sentiment, and more to set capacity in the moment.
• Give you deep—and actionable—insight into your customers’ motivations and concerns through both real-time and historical AI-driven analysis of every conversation.

‍

How does 3X productivity sound?

Helping agents helps companies achieve breakthrough results.

We’ve seen it time and again at ASAPP as our customers double and triple productivity—yes, really—and at the same time increase customer satisfaction scores and build brand loyalty.

That’s something no legacy chat technology can do.

Let us show you real results—in less than 60 days.

When talking to Contact Center leaders, we constantly hear the refrain, “How do I bring down handle times?” As long as contact centers have been around, the standard measure of agent efficiency has been how quickly they can handle a customer’s inquiry and move on to the next customer. In the phone world, the simplest way to improve the output of a phone agent is to shorten their call time. Thus handle time (or AHT) has long been our key metric.

With the onset of digital channels, and increasing adoption of these channels by both customers and companies, handle time is no longer an adequate measure of agent efficiency.

As a Success Manager at ASAPP, I work with our customers to drive value and optimize contact center operations for digital and voice programs. A key part of our effort is identifying the right metrics for analysis to measure program success. We are continuously evolving what we measure and the recommendations made based on changes in the way companies interact with customers (over multiple channels and sometimes asynchronously, for example.) And, on the specific goals of the individual company.

In order to best evaluate program performance, particularly agent efficiency, we need to look beyond AHT for three key reasons:

1. Digital automation is changing the average conversation that gets to a live agent. The simplest customer inquiries are able to be solved by automation, and even those issues that used to be too complex for automation can often be solved through integrations to company systems. For example, an internet service customer can check if they are in an outage, troubleshoot their modem, send diagnostic information, AND schedule a service appointment, without ever speaking to a live agent. This expansion of automation causes only the most complex issues to get to agents, driving up the average time it takes to handle those inquiries.
2. Digital agents may have a more expansive set of tools than ever before. It’s not just about being able to handle more than one chat at a time, digital agents can rely on automation and augmentation to maximize the number of customers they handle at once.
3. Voice and digital conversations just aren’t the same, and we need to identify a way to benchmark them at a programmatic level. The handle time of a voice call is all of the time the agent and the customer are spending on that one call, while the handle time for a digital conversation is all of the time the customer is on that conversation. Digital agents are likely to be handling multiple conversations concurrently. And, a digital conversation may stop and start over multiple sessions, as customers handle their inquiry asynchronously.

Customer Experience vs. Agent Efficiency

This isn’t to say that AHT is irrelevant, in fact it is still very relevant, but not in regards to agent performance. AHT is most relevant for the customer experience. Companies must still be concerned with how long the experience is for a customer to handle their issue. What I have started pushing for is the shift in perspective, AHT is a KPI for the customer experience, but when evaluating front line agent efficiency/output, we have better measures.

From AHT to Throughput

In order to find the best opportunities to maximize workforce efficiency, modern CX teams have shifted focus from AHT to Throughput. Throughput can be measured a number of different ways, but simply it is a measure of an agent’s output in a certain time period, normally an hour.

‍

Throughput measures an agent’s output in a way that works for both voice and digital engagement, including asynchronous interactions.

Stephen Canterbury

‍

In most cases, organizations are measuring this as resolutions/assignments per utilized/logged-in hour. This measure can easily be translated into a simplified cost per chat metric and overall, gives a holistic picture of how much can one front line team member do. Throughput also helps to avoid the biases of handle time, because it can be based off of total time that an agent is working, potentially highlighting absenteeism or other addressable gaps.

Take the below example of a way that we are seeing organizations start to shift their thinking:

Agent A: On the clock for 5 hours, handles 20 customer issues, Average Handle Time = 20 minutes.

Agent B: On the clock for 5 hours, handles 25 customer issues, Average Handle Time = 20 minutes

Assuming both agents are working the same shift, we would obviously want Agent B’s performance over Agent A’s. Agent B will handle an assignment more per hour than Agent A, while the customer experiences the same handle time. Combining this analysis with Occupancy and Customer Satisfaction enable an organization to get a complete picture of the body of work of an agent. Throughput becomes the measure to assess efficiency. Those agents that can handle more customers per hour, while staying within CSAT and business (occupancy) targets, are the top performers. We can use their behaviors as a template to better train and support the rest of the agent base.

Where do we go from here?

Technology advancements are continuing to push the contact center industry forward. Automation is becoming easier to build and integrate with company systems. Only the most complex customer issues will need live support. Artificial Intelligence is getting smarter and more helpful in supporting live agent conversations. The lines between live agent and automation blur as agents are supported by recommendations, automated flows, and more. Throughput will be a metric that can scale with the changing landscape, better than any other measure of agent efficiency.

Efficiency at 30,000ft—Organizational Throughput

Even still, some forward-thinking organizations are looking beyond agent throughput to a broader program efficiency view. The question we are starting to hear is “How efficient is my Customer Experience program?” Companies are leaning into throughput, but viewing it in a programmatic lens. When you combine both the automated and live aspects of your program, how effective is your investment? Program effectiveness is being measured when looking at all automated AND handled customers per staffed agent.

Organizational Throughput is helping to view the program as a whole, tracking improvements or degradations in automation and live support on the same playing field. As the worlds of automation and live support become more intertwined, it only makes sense for organizations to start looking at these two separate entities’ performance together.

For many years, companies have measured the effectiveness of automated systems, such as their chatbot or IVR, by the system’s Containment Rate—the percent of interactions that don’t reach a human agent.

For digital chat programs, optimizing a bot to increase Containment Rate certainly has benefits. If some customers have their problems resolved in fully automated experiences without engaging an agent, then agents’ time will be freed up to assist other customers. Solving more customers’ issues in the bot means companies may require fewer employees to handle chat volume, resulting in cost savings.

What containment rate doesn’t measure

The problem with only measuring Containment Rate is that deflecting a customer doesn’t mean they’ve had their issue resolved. It simply means that a digital agent didn’t get involved in that particular interaction.

In my role as a Customer Experience Strategist at ASAPP, my team and I work with our customers to ensure that their customers are receiving the best possible experience when interacting with the brand. To optimize the customer’s experience, we need to ensure the metrics we are tracking and measuring are answering the key question “Did I resolve my customer’s issue?”

At ASAPP, when judging the effectiveness of a bot and making decisions to improve that effectiveness, we recommend using a metric called Flow Success—the number of conversations where the customer was provided with information necessary to address their issue without the need for a rep to get involved. Using this metric enables companies to understand when their containment is “good containment” and unlocks additional opportunities to optimize their bots towards a great experience.

Why flow success?

It is possible for a chatbot to have a high Containment Rate but a low Flow Success Rate. While this may represent potential cost savings for the company, this is an extremely frustrating experience for the end user.

Some automated flows require customers to take multiple, sometimes unnecessary, steps to find the solution to their problem. Other times, customers may be forced to log in to their account before they can get information when the solution could be provided to them on the phone without logging in. Sometimes customers may choose the wrong path in a flow and give up when they get information that isn’t relevant. These are all examples of “bad containment,” counting towards high Containment but low Flow Success.

In a best case scenario, the customer abandons the experience because they found the answer to their question elsewhere. However, there is a greater risk that the customer gets frustrated with their bot experience and calls instead, forcing involvement from a voice agent, ultimately increasing the cost to resolve that issue. Even worse, a customer may become so annoyed that they become a churn risk for the company. The loss in customer lifetime value can greatly outweigh the cost of having that customer interact with a digital agent.

When shifting the focus from Containment Rate to Flow Success, we are able to help our customers identify and fix areas where this may be happening.

‍

It’s important for CX teams to understand not just whether their automation contained the customer, but whether the customer’s need was actually served.

Bobby Kovalsky

‍

For example, when we analyzed a US cable company’s virtual assistant experience, we found a large gap in Containment and Flow Success for a billing intent. Customers asking to have their bill explained were often contained within the bot but were rarely provided with the information they wanted about their bill. Further analysis revealed that customers were frustrated by the amount of information they needed to provide the bot before the bot would give them their answer. To improve the experience, we recommended the company remove some of these steps, which we ultimately were unnecessary in determining the response the customer needed.

After the company implemented our optimization recommendation, the automated flow not only saw improved Flow Success but it ultimately led to greater Containment. The share of conversations with this intent that were Contained without Flow Success decreased by 21%. Because customers were easily able to access their information in the bot experience, they were less likely to ask to speak to an agent, leading to a 29% increase in Containment Rate.

What are the trade offs?

We’ve seen that organizations that focus on Flow Success rather than strictly Containment are able to create a better customer experience. However, this sometimes means customers will be able to more easily reach a representative. Increasing “good containment” and reducing “bad containment” does not always correlate with an increase in overall Containment.

For example, an internet service provider saw high levels of Containment when customers were asking if they were in an outage. After conducting an analysis to identify areas where customers were not being told whether or not they were in an outage, we found that the existing authentication process was causing customers to abandon the bot. We recommended the company revisit their existing process.

By simplifying the sign-in process, the company ultimately made it easier for customers to reach digital reps and Containment decreased by 3%. However, significantly more customers were informed about the status of their outage, leading to a 17% increase in Flow Success. This organization accepted the tradeoff, allowing more customers to reach digital agents knowing that the large dropoff from the previous sign-in process contributed to increased call volume and therefore higher overall costs.

Although these types of changes may lead to lower Containment, they will ultimately drive higher organizational throughput. By enhancing the digital experience, customers will be more likely to choose digital channels for their future contacts. As the contact mix shifts towards digital, companies unlock additional benefits unique to the channel, such as increasing concurrency, which enable them to handle more conversations with fewer representatives. This leads to larger cost savings than they would have achieved by preventing customers from reaching a digital rep.

What else should be considered?

Measuring Flow Success helps companies analyze and optimize their bot but it is not the only metric that matters. Companies may also want to consider the bot’s contribution to first contact resolution, call prevention, and customer satisfaction.

Creating the best bot experience requires companies to continuously evaluate and optimize performance. Those who focus on delivering the best customer experience in the bot rather than just lowering costs see long term benefits through increased customer satisfaction and higher digital adoption.

It doesn’t take a rocket scientist (or a machine learning engineer) to know that customer service representatives need the right tools to do their job. If they don’t have them, both rep and customer satisfaction suffer.

As a Senior Customer Experience manager at ASAPP, I’ve spent the last several years partnering with some of the world’s largest consumer companies to improve their contact center operations and overall customer experience. As they adopt our technology, they’re eager to see the investment’s impact on operational performance. Measuring success typically starts with examining how agents interact with the system.

What happens when we empower agents with better technology? Do we create a positive feedback loop of happier employees and more satisfied customers? How can we track and learn from our best performers? Can usage be correlated with other important agent efficiency and success metrics? These are the types of questions my team seeks to answer with our customers as we evaluate a program’s augmentation rate.

Our suite of integrated automation features, including AutoSuggest, AutoComplete, and AutoPilot, use machine learning to augment agent activity. The system recommends what reps should say or do during the course of an interaction with a customer. The machine learning models improve with usage—which in the contact center space can be millions of interactions per month. We work with our customers to measure the impact of these technologies on their operations and KPIs through our augmentation rate metric, which evaluates the percentage of messages sent by agents that were suggested by our algorithms.

A recent analysis found that each time one of our customers’ agents used a suggested response instead of typing freehand, they saved ~15 seconds. The time savings added up fast.

Jonathan Rossi

Augmentation rate isn’t a common metric (yet). But it offers tremendous value as an indicator of how well the technology is being adopted, and therefore, the likelihood it will have an impact on performance outcomes.

From my experience, the top three things operators should know when utilizing this metric are:

1. Iteration over time:
2. Augmentation rate offers guidance on:
3. How well the system is augmenting agent responses and learning through data;
4. How well reps are trained and coached to use the tools available to them inside our platform.
5. Both the system’s model and rep training can be calibrated and optimized to continually increase the effectiveness of these features.
6. Workforce Management (WFM) Implications:
7. The top-level augmentation metric is helpful in measuring overall program health, but looking at usage across groups and individuals can also be extremely informative for supervisors and coaches when assessing agent and cohort performance.
8. We’ve found correlations between increased augmentation usage, AHT reduction, and improved CSAT for high-performing reps.
9. Incentives matter.
10. If you incentivize a workforce on this metric alone, there can be adverse effects. We’ve seen reps attempt to “game the system” by always using a suggested message, then editing the response before sending. This actually increases conversation duration and decreases productivity compared to not using the features in the first place.
11. Augmentation should be one of multiple metrics that go into agent performance incentives (alongside others like CSAT, throughput, and resolution rate).

By studying augmentation rates at customer companies, we’ve been able to see exactly where agents get the most benefit from integrated automation and where pain points still exist. From that knowledge, ASAPP has begun building new capabilities to increase the impact ML can have on modern workforces. For example:

• Our product team is developing additional AutoPilot features (like AutoPilot Greetings) that will automate the beginning of conversations, so reps can focus on the “meat” of an interaction and better assist the customer.
• We know that both agents and customers prefer personalized conversations. Our research and product teams are tackling this problem in two ways. First, we incorporate custom responses into our platform, enablinging reps to curate a repository of preferred messages to send to customers. This allows for agents to use suggestions in their own voice. Second, as we get more malleable in leveraging customer-specific data throughout our platform, we’re embedding more personalized customer information directly into these suggestions.

‍

Early feedback on these additions to our augmentation features have been overwhelmingly positive from both agents and operators. Like our machine learning models, we aim to iteratively improve our product capabilities over time through usage and impact analysis, working with our customers to radically increase rep satisfaction and efficiency—which ultimately benefits the customer experience.

Automating common tasks and enabling self-service issue resolution for customers is an essential part of any online customer service experience. These automated flows directly address a specific well-scoped problem for the customer, getting them to resolution quicker and freeing up agents to handle more complex issues. But, automation doesn’t have to be an all or nothing proposition. At ASAPP, we automate flows before, during, and after agent interactions, increasingly reducing agent workload and growing the opportunity for self service over time.

Discovering and prioritizing new flows and understanding what’s needed for successful automation, however, can be challenging. It is often a time consuming and labor intensive process. ASAPP has developed AI Native® approaches to surface these workflows to humans, and we’ve been awarded a patent, “Identifying Representative Conversations Using a State Model” for a powerful solution we developed to perform flow induction.

It’s difficult for a human to imagine all the possible conversation patterns that could be automated, and which ones are most important to automate. It’s important to consider things like how many users it would affect, how much agent time is being spent on the intent, whether the flow has a few well-defined paths or patterns, what value the intent brings to the business, and whether there are any overlaps between this intent and other conversations.

‍

Rather than manually sifting through all the data, an analyst can leverage patterns identified by the model to more quickly deploy automated workflows and evaluate their potential with real usage data.

Michael Griffiths

‍

We call the process of automatically discovering and distilling the conversational patterns—“workflows”, or “flows” for short—flow induction. We can condense a large collection of possible flows to a much smaller number of representative flows. These induced flows best capture interactions between customers and agents, and flags where automation can lend a helping hand. This facilitates faster and more comprehensive creation of automated flows, saving time and money.

Our patented approach for flow induction begins by representing each part of a conversation mathematically, capturing its state at the time. As a simple example, we would want the start of each conversation—where agents say “hello” or “how are you” or “welcome to X company”—to be similar, with approximately the same state representation. We can then trace the path the conversation traces as it progresses from start to finish. If the state is two dimensional, you could draw the line that each conversation takes as its own “journey.” We then group similar paths and identify recurring patterns within and across conversations.

The process of identifying automation use cases is dramatically simplified with this representation. Instead of manually sifting through conversations, talking to experienced agents, or listening into calls to do journey mapping—the analyst can dive into a pattern the model has identified and review its suitability for automation. Even better, because ASAPP is analyzing every customer interaction, we know how many customers are affected by the flows and what the outcomes (callback, sales conversion, etc) are — making prioritization a breeze.

ASAPP deploys “flows” like this across our platform. By identifying the recurring work that agents are handling an analyst can construct integrated flows for agents to serve in any part of a conversation. And over time, more and more flows can be sent directly to the customer so they can self-serve. Once deployed every flow becomes part of a virtuous feedback loop, where usage informs how impactful the automation is for our customers and their customers. This process informs both new flow opportunities and refinements to existing flows.