meta data for this page
  •  

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
logging_strategy [2018/06/13 12:12]
miket_forty7ronin.com 		logging_strategy [2019/08/08 10:43] (current)
lisa.illgen_concentrix.com [Call containment]
Line 1: Line 1:
 ====== Logging Strategy ====== ====== Logging Strategy ======
  
-**Why logging is important**\\ +==== Why logging is important ​==== 
-Comprehensive logging provides the foundation for comprehensive reporting. Logging and reporting are key to tuning and problem identification,​ both at launch by the company implementing the system and long-term by the business itself. Think about what you will need to be able to evaluate the system effectively and on an ongoing basis. This will help you determine whether you only need a system'​s built-in features, or whether you might also need custom code and/or whole-call recording.+ ​Comprehensive logging provides the foundation for comprehensive reporting. Logging and reporting are key to tuning and problem identification,​ both at launch by the company implementing the system and long-term by the business itself. Think about what you will need to be able to evaluate the system effectively and on an ongoing basis. This will help you determine whether you only need a system'​s built-in features, or whether you might also need custom code and/or whole-call recording.
  
-**Waterfall report**\\ +==== Waterfall report ​==== 
-VUI designers often find waterfall reports especially useful. A waterfall report provides information about how call volumes flow through the different possible paths of an IVR, and ideally provide information for each dialog step about the percentage of calls: + VUI designers often find waterfall reports especially useful. A waterfall report provides information about how call volumes flow through the different possible paths of an IVR, and ideally provide information for each dialog step about the percentage of calls: 
-\\ \\  ​'​monospaced' ​• Transferred to an agent (and ideally, which skill group received the call, whether a transfer at this point is expected or unexpected, and whether the transfer was caller- or system-initiated)+\\ \\  • Transferred to an agent (and ideally, which skill group received the call, whether a transfer at this point is expected or unexpected, and whether the transfer was caller- or system-initiated)
 \\ • Disconnected (and ideally, whether a caller disconnection at this point is expected or unexpected and caller- or system-initiated) \\ • Disconnected (and ideally, whether a caller disconnection at this point is expected or unexpected and caller- or system-initiated)
 \\ • In which callers experienced a noinput event \\ • In which callers experienced a noinput event
Line 17: Line 17:
 Without this level of logging detail, it is very difficult after deployment to identify hot (problem) spots in the flow, which makes it difficult to provide detailed recommendations for improvement. Without this level of logging detail, it is very difficult after deployment to identify hot (problem) spots in the flow, which makes it difficult to provide detailed recommendations for improvement.
  
-**Common metrics**\\ +==== Common metrics ​==== 
-Common metrics, ideally provided both as global measures and for each dialog step (e.g., in a waterfall report), include:+ ​Common metrics, ideally provided both as global measures and for each dialog step (e.g., in a waterfall report), include:
 • Abandon rate • Abandon rate
 • Call containment • Call containment
Line 25: Line 25:
 They each have strengths and weaknesses as metrics. They each have strengths and weaknesses as metrics.
  
-Abandon rate+==== Abandon rate ==== 
 The abandon rate is the percentage of callers who hang up on the application early on. Defining "early on" is the first challenge of this metric. Secondly, it isn't always easy to interpret the abandon rate. What does it mean if a caller hangs up during an IVR's introduction?​ What if a system provides a caller'​s account balance up-front, then the caller hangs up during the following main menu? Is it more likely that these callers heard what they needed and disconnected,​ or might they have been confused or irritated by something in the main menu?  The abandon rate is the percentage of callers who hang up on the application early on. Defining "early on" is the first challenge of this metric. Secondly, it isn't always easy to interpret the abandon rate. What does it mean if a caller hangs up during an IVR's introduction?​ What if a system provides a caller'​s account balance up-front, then the caller hangs up during the following main menu? Is it more likely that these callers heard what they needed and disconnected,​ or might they have been confused or irritated by something in the main menu? 
  
 Although it is not included in built-in logging, it can be helpful for later analysis and interpretation to define abandoned calls at each dialog step as expected (e.g., due to having just completed a task or heard a key bit of information) or unexpected (no compelling reason for the caller to have disconnected). Although it is not included in built-in logging, it can be helpful for later analysis and interpretation to define abandoned calls at each dialog step as expected (e.g., due to having just completed a task or heard a key bit of information) or unexpected (no compelling reason for the caller to have disconnected).
  
-Call containment +==== Call containment ​==== 
-Don't put inappropriate emphasis on call containment +** // Don't put inappropriate emphasis on call containment ​// ** 
-Call containment refers to the percentage of calls that began and ended in the IVR -- no transfer to an agent (Bloom et al., 2005). Other terms for this metric include calls serviced in the IVR, self-service resolution, IVR utilization,​ IVR take-rate, and call retention. (Note that abandoned calls also satisfy this definition, so be sure you've defined them each clearly for your system.)+\\ Call containment refers to the percentage of calls that began and ended in the IVR -- no transfer to an agent ([[references#​bloom2005|Bloom et al., 2005]]). Other terms for this metric include calls serviced in the IVR, self-service resolution, IVR utilization,​ IVR take-rate, and call retention. (Note that abandoned calls also satisfy this definition, so be sure you've defined them each clearly for your system.)
  
-This is a common and widely-tracked metric, but it is deeply flawed. In particular, it ignores an important purpose of most IVRs -- to route callers to the correct skill group -- and also disregards the potential benefits of partial automation (Suhm, 2008; also see Partial Automation vs. Full Automation). As Leppik (2006, p. 1) stated:+This is a common and widely-tracked metric, but it is deeply flawed. In particular, it ignores an important purpose of most IVRs -- to route callers to the correct skill group -- and also disregards the potential benefits of partial automation ([[references#​suhm2008|Suhm, 2008]]; also see [[Partial Automation vs. Full Automation)]]. As [[references#​leppik2006|Leppik]] (2006, p. 1) stated:
  
 "​There’s a couple of unstated assumptions built into this metric which make it a poor way to measure IVR performance. The first assumption is that the system’s only function is to automate customer calls. In truth, the IVR has a second, just as important (and maybe more important) function: to identify which customers’ calls must go to an agent, and efficiently connect those customers to people who can help them. This latter group of calls is going to include the sales calls, billing errors, and already-upset customers who have been trying to resolve their problem for weeks. You can never automate those calls, and failing to identify them and get them off the IVR and into an agent’s hands is as much of a system failure as sending a potential self-service call to a human."​ "​There’s a couple of unstated assumptions built into this metric which make it a poor way to measure IVR performance. The first assumption is that the system’s only function is to automate customer calls. In truth, the IVR has a second, just as important (and maybe more important) function: to identify which customers’ calls must go to an agent, and efficiently connect those customers to people who can help them. This latter group of calls is going to include the sales calls, billing errors, and already-upset customers who have been trying to resolve their problem for weeks. You can never automate those calls, and failing to identify them and get them off the IVR and into an agent’s hands is as much of a system failure as sending a potential self-service call to a human."​
  
-Be wary of quantified expectations of call containment +** //Be wary of quantified expectations of call containment ​// ** 
-What kind of containment a system should achieve is going to be largely driven by what the capabilities are, and why people call. If the capabilities of the IVR encompass only 10% of why people call in, then the stretch goal would be 10% call containment (because we are never going to serve 100% of the people who could be served in the IVR). The other thing to be wary of is the fact that both systems and caller goals change with time. Let's look at an example.+\\ What kind of containment a system should achieve is going to be largely driven by what the capabilities are, and why people call. If the capabilities of the IVR encompass only 10% of why people call in, then the stretch goal would be 10% call containment (because we are never going to serve 100% of the people who could be served in the IVR). The other thing to be wary of is the fact that both systems and caller goals change with time. Let's look at an example.
  
 Suppose a travel company mainly fields calls for problems, things that can't be done on the website, change in travel plans, etc. Normal containment is 25%. Now let's say a blizzard hits the entire northeastern U.S. Lots of plans are disrupted, and there'​s nothing the automated service can do about it. Containment during this time period will plummet, and nobody should be penalized for it. Now let's say things are back to normal, and there'​s a decision in the business to make sure you get the sale, be it online or in the call center, so the "​contact us" phone number is placed much more prominently throughout the site, driving calls up to the IVR which can't be automated. This will lead to an ongoing drop in containment. Suppose a travel company mainly fields calls for problems, things that can't be done on the website, change in travel plans, etc. Normal containment is 25%. Now let's say a blizzard hits the entire northeastern U.S. Lots of plans are disrupted, and there'​s nothing the automated service can do about it. Containment during this time period will plummet, and nobody should be penalized for it. Now let's say things are back to normal, and there'​s a decision in the business to make sure you get the sale, be it online or in the call center, so the "​contact us" phone number is placed much more prominently throughout the site, driving calls up to the IVR which can't be automated. This will lead to an ongoing drop in containment.
Line 45: Line 46:
 What you should be seeing is that "​containment"​ is a highly charged word, often times the wrong goal, and sometimes subject to forces completely outside a designer'​s control. What you should be seeing is that "​containment"​ is a highly charged word, often times the wrong goal, and sometimes subject to forces completely outside a designer'​s control.
  
-As another example, Suhm (2008, p. 20) warned:+As another example, ​[[references#​suhm2008|Suhm]] (2008, p. 20) warned:
  
 "While often interpreted as the success rate for serving callers in an automated fashion, IVR take-rate is a poor measure of the effectiveness of an IVR, because callers hanging up in the IVR may not have received any useful information. In several large call centers we have seen that the majority of callers hanging up have actually received no useful information and therefore have not been served. For example, based on standard IVR reports, one call center believed that its IVR served more than 30% of the callers in the automated system. A detailed analysis based on end-to-end calls revealed that only 2% of all callers were actually served. Almost 20% hung up without receiving any useful information,​ and some 8% hung up while on hold for an agent."​ "While often interpreted as the success rate for serving callers in an automated fashion, IVR take-rate is a poor measure of the effectiveness of an IVR, because callers hanging up in the IVR may not have received any useful information. In several large call centers we have seen that the majority of callers hanging up have actually received no useful information and therefore have not been served. For example, based on standard IVR reports, one call center believed that its IVR served more than 30% of the callers in the automated system. A detailed analysis based on end-to-end calls revealed that only 2% of all callers were actually served. Almost 20% hung up without receiving any useful information,​ and some 8% hung up while on hold for an agent."​
Line 51: Line 52:
 The converse of call containment is the transfer rate, which, like abandonment can be difficult to interpret without having defined in advance the dialog steps at which you expect calls to leave the IVR (which will typically require custom coding, but will make the data much easier to interpret down the road). The converse of call containment is the transfer rate, which, like abandonment can be difficult to interpret without having defined in advance the dialog steps at which you expect calls to leave the IVR (which will typically require custom coding, but will make the data much easier to interpret down the road).
  
-First call resolution+==== First call resolution ​====
 First call resolution refers to the ability of a call center to resolve customer requests on the first try with no transfers and no callbacks. Note that it is important to specify how FCR is measured. Just because a caller calls back within X hours does not by itself indicate an FCR failure -- appropriate measurement methods need to factor in the call reason both at the IVR and agent level. First call resolution refers to the ability of a call center to resolve customer requests on the first try with no transfers and no callbacks. Note that it is important to specify how FCR is measured. Just because a caller calls back within X hours does not by itself indicate an FCR failure -- appropriate measurement methods need to factor in the call reason both at the IVR and agent level.
  
 Measuring first call resolution requires an architecture that has a centralized end-to-end logging and reporting system. It can be expensive to implement (especially if the architecture doesn'​t already exist) but can be a valuable measure. Measuring first call resolution requires an architecture that has a centralized end-to-end logging and reporting system. It can be expensive to implement (especially if the architecture doesn'​t already exist) but can be a valuable measure.
  
-References+==== References ​====
 Bloom, J., Gilbert, J. E., Houwing, T., Hura, S., Issar, S., Kaiser, L., et al. (2005). Ten criteria for measuring effective voice user interfaces. Speech Technology, 10(9), 31–35. Bloom, J., Gilbert, J. E., Houwing, T., Hura, S., Issar, S., Kaiser, L., et al. (2005). Ten criteria for measuring effective voice user interfaces. Speech Technology, 10(9), 31–35.