Enable end-to-end call recording if at all possible
When it comes to getting both quantitative and qualitative information to guide iterative improvement of an IVR, there is no substitute for end-to-end call recording – providing a recording of everything the caller hears and does when dealing with the IVR and, if transferred, with the call center (hold time, hold experience, agent-caller interaction). The resulting data is powerfully valid because it is based on real calls made by real users trying to do real tasks. As Lewis (2011, p. 122) stated:

“Data doesn’t get any better than that. Even though the sample size of the data (the number of calls) in call logs will be substantially greater than the number of transcribed end-to-end calls, call logs cannot provide the richness of knowing exactly where in a prompt or message a caller barged in, or exactly where a poorly timed message caused the caller to stutter, resulting in a misrecognition.”

Ideally, the calls are recorded end-to-end, but even recordings that only capture caller interactions with the IVR can be valuable in guiding redesign.

Larger sample sizes are generally preferable, but it is possible to get substantial information from as few as 100 calls (Cohen et al., 2004).

In addition to enhancing the discovery and resolution of usability problems, the analysis of end-to-end calls can also provide (Alwan & Suhm, 2010; Lewis, 2011; Suhm, 2008):

• The distribution of different types of calls
• Call-path analysis
• Correct transfer rate
• Total IVR benefit (an estimate of the average amount of agent time saved by the IVR)

Keep in mind that when you're using small sample sets, the resultant data is qualitative. The safest approach to leveraging call recording findings is to use analytics, or some other method based on logging data, to assess the true impact of the issues uncovered by listening to call recordings. Whether and how to address those uncovered issues then can and should have the true quanititative impacts factored into the get-well plan.

Alwan, J., & Suhm, B. (2010). Beyond best practices: A data-driven approach to maximizing self-service. In W. Meisel (Ed.), Speech in the user interface: Lessons from experience (pp. 99–105). Victoria, Canada: TMA Associates.

Cohen, M. H., Giangola, J. P., & Balogh, J. (2004). Voice user interface design. Boston, MA: Addison-Wesley.

Lewis, J. R. (2011). Practical speech user interface design. Boca Raton, FL: CRC Press, Taylor & Francis Group.

Suhm, B. (2008). IVR usability engineering using guidelines and analyses of end-to-end calls. In D. Gardner-Bonneau & H. E. Blanchard (Eds.), Human factors and voice interactive systems, 2nd edition (pp. 1-41). New York, NY: Springer.