Comprehensive logging provides the foundation for comprehensive reporting. Logging and reporting are key to tuning and problem identification, both at launch by the company implementing the system and long-term by the business itself. Think about what you will need to be able to evaluate the system effectively and on an ongoing basis. This will help you determine whether you only need a system's built-in features, or whether you might also need custom code and/or whole-call recording.
VUI designers often find waterfall reports especially useful. A waterfall report provides information about how call volumes flow through the different possible paths of an IVR, and ideally provide information for each dialog step about the percentage of calls:
• Transferred to an agent (and ideally, which skill group received the call, whether a transfer at this point is expected or unexpected, and whether the transfer was caller- or system-initiated)
• Disconnected (and ideally, whether a caller disconnection at this point is expected or unexpected and caller- or system-initiated)
• In which callers experienced a noinput event
• In which callers experienced a nomatch event
• Which went to the next step(s) in the callflow
In addition to these percentages, it is helpful to know how long (average and standard deviation) it took for the caller to get to the end of this step.
If your system's built-in logging does not provide the level of detail of a waterfall report, you should plan on custom coding to get it Without this level of logging detail, it is very difficult after deployment to identify hot (problem) spots in the flow, which makes it difficult to provide detailed recommendations for improvement.
Common metrics, ideally provided both as global measures and for each dialog step (e.g., in a waterfall report), include: • Abandon rate • Call containment • First call resolution
They each have strengths and weaknesses as metrics.
The abandon rate is the percentage of callers who hang up on the application early on. Defining “early on” is the first challenge of this metric. Secondly, it isn't always easy to interpret the abandon rate. What does it mean if a caller hangs up during an IVR's introduction? What if a system provides a caller's account balance up-front, then the caller hangs up during the following main menu? Is it more likely that these callers heard what they needed and disconnected, or might they have been confused or irritated by something in the main menu?
Although it is not included in built-in logging, it can be helpful for later analysis and interpretation to define abandoned calls at each dialog step as expected (e.g., due to having just completed a task or heard a key bit of information) or unexpected (no compelling reason for the caller to have disconnected).
Don't put inappropriate emphasis on call containment
Call containment refers to the percentage of calls that began and ended in the IVR – no transfer to an agent (Bloom et al., 2005). Other terms for this metric include calls serviced in the IVR, self-service resolution, IVR utilization, IVR take-rate, and call retention. (Note that abandoned calls also satisfy this definition, so be sure you've defined them each clearly for your system.)
This is a common and widely-tracked metric, but it is deeply flawed. In particular, it ignores an important purpose of most IVRs – to route callers to the correct skill group – and also disregards the potential benefits of partial automation (Suhm, 2008; also see Partial Automation vs. Full Automation). As Leppik (2006, p. 1) stated:
“There’s a couple of unstated assumptions built into this metric which make it a poor way to measure IVR performance. The first assumption is that the system’s only function is to automate customer calls. In truth, the IVR has a second, just as important (and maybe more important) function: to identify which customers’ calls must go to an agent, and efficiently connect those customers to people who can help them. This latter group of calls is going to include the sales calls, billing errors, and already-upset customers who have been trying to resolve their problem for weeks. You can never automate those calls, and failing to identify them and get them off the IVR and into an agent’s hands is as much of a system failure as sending a potential self-service call to a human.”
Be wary of quantified expectations of call containment
What kind of containment a system should achieve is going to be largely driven by what the capabilities are, and why people call. If the capabilities of the IVR encompass only 10% of why people call in, then the stretch goal would be 10% call containment (because we are never going to serve 100% of the people who could be served in the IVR). The other thing to be wary of is the fact that both systems and caller goals change with time. Let's look at an example.
Suppose a travel company mainly fields calls for problems, things that can't be done on the website, change in travel plans, etc. Normal containment is 25%. Now let's say a blizzard hits the entire northeastern U.S. Lots of plans are disrupted, and there's nothing the automated service can do about it. Containment during this time period will plummet, and nobody should be penalized for it. Now let's say things are back to normal, and there's a decision in the business to make sure you get the sale, be it online or in the call center, so the “contact us” phone number is placed much more prominently throughout the site, driving calls up to the IVR which can't be automated. This will lead to an ongoing drop in containment.
What you should be seeing is that “containment” is a highly charged word, often times the wrong goal, and sometimes subject to forces completely outside a designer's control.
As another example, Suhm (2008, p. 20) warned:
“While often interpreted as the success rate for serving callers in an automated fashion, IVR take-rate is a poor measure of the effectiveness of an IVR, because callers hanging up in the IVR may not have received any useful information. In several large call centers we have seen that the majority of callers hanging up have actually received no useful information and therefore have not been served. For example, based on standard IVR reports, one call center believed that its IVR served more than 30% of the callers in the automated system. A detailed analysis based on end-to-end calls revealed that only 2% of all callers were actually served. Almost 20% hung up without receiving any useful information, and some 8% hung up while on hold for an agent.”
The converse of call containment is the transfer rate, which, like abandonment can be difficult to interpret without having defined in advance the dialog steps at which you expect calls to leave the IVR (which will typically require custom coding, but will make the data much easier to interpret down the road).
First call resolution refers to the ability of a call center to resolve customer requests on the first try with no transfers and no callbacks. Note that it is important to specify how FCR is measured. Just because a caller calls back within X hours does not by itself indicate an FCR failure – appropriate measurement methods need to factor in the call reason both at the IVR and agent level.
Measuring first call resolution requires an architecture that has a centralized end-to-end logging and reporting system. It can be expensive to implement (especially if the architecture doesn't already exist) but can be a valuable measure.
Bloom, J., Gilbert, J. E., Houwing, T., Hura, S., Issar, S., Kaiser, L., et al. (2005). Ten criteria for measuring effective voice user interfaces. Speech Technology, 10(9), 31–35.
Leppik, P. (2006). Developing metrics part 1: Bad metrics. The Customer Service Survey. Retrieved from www.vocalabs.com/resources/blog/C834959743/E20061205170807/index.html.
Suhm, B. (2008). IVR usability engineering using guidelines and analyses of end-to-end calls. In D. Gardner-Bonneau & H. E. Blanchard (Eds.), Human factors and voice interactive systems, 2nd edition (pp. 1-41). New York, NY: Springer.