Hardly anything is more annoying than a recognition system that doesn't remember caller rejections during confirmation, for example:

• System: From which account?

• Caller: My checking account.

• System: Was that your checking account?

• Caller: Yes.

• System: And how much do you want to transfer?

• Caller: $500.

• System: Was that $900?

• Caller: No, $500.

• System: Was that $900?

• Caller: NO!!!!

To avoid this situation, use a skip list as part of your confirmation strategy. When the recognizer received corrected input, the application compares the new n-best list (list of recognition candidates ordered by confidence) to the skip list and, if the highest-confidence item in the n-best list is in the skip list, it skips to the item with the next-highest confidence.

The idea of a skip list goes back to Ainsworth and Pratt (1992, 1993), who found that using a skip list (their “repetition with elimination” strategy) reduced the number of correction steps required to complete a task.

In a discussion group at SpeechTek 2013, a potential problem with a simple skip list was raised. Sometimes callers will reject what they actually wanted, and if that gets put into a skip list, then the application will no longer receive that item from the recognizer. A possible solution to this is to modify the skip list programming so rather than absolutely preventing an item in the skip list from being reused, the item(s) in the skip list are penalized to some extent. Thus, if the acoustic evidence is just overwhelming, an item in the skip list might be offered to the user again. If, however, the item is in a skip list and just barely beats the next item in the n-best list, that penalty will knock it down in the list. The exact amount of the penalty can be tuned to minimize, but probably not completely eliminate, presentation errors. This is a more complex method, but it addresses the complexity of the real world – a world in which callers will be angry if an application seems to keep recognizing something even though they have rejected it, but in which some callers will accidentally reject something they actually wanted.

Also see Using n-best Lists and Advanced confirmation and error correction with confidence levels and n-best lists.

Ainsworth, W. A., & Pratt, S. R. (1992). Feedback strategies for error correction in speech recognition systems. International Journal of Man-Machine Studies, 36, 833–842.

Ainsworth, W. A., & Pratt, S. R. (1993). Comparing error correction strategies in speech recognition systems. In C. Baber & J. M. Noyes (Eds.), Interactive speech technology: Human factors issues in the application of speech input/output to computers (pp. 131–135). London, UK: Taylor & Francis.