Different types of impairment indicate different design modifications.

Callers with Hearing Impairments
There are two main points to remember when dealing with the hearing impairment population: avoid high-pitched voice talents (see Voice Talent Selection section), and TDD and TYY systems are a must (see TDD section).

Callers with Speech Impairments
All speech applications must have a touchtone fallback. Moreover, it's extremely useful when the caller can guess the touchtone options without having to wait for the first error message (that is, the options are read with distinct gaps so the caller can pretty much guess that if she presses 1 she'll go to the first option). Since these users will likely NOT prefer to transfer to a live person, maximizing automation available in touchtone will be appreciated. It would also be nice to remember the caller's preference and keep on the touchtone path if touchtone is used initially, although this is more of a usability thing.

Callers with Motor Impairments
The only consideration here is the additional time to respond as mentioned in the hearing impaired/TDD section. This is a vast set of users who may be using anything from a head pointer to an eye tracking system but again will likely have assistive devices that allow for fairly accurate use. The touchtone equivalent may help most though, as these are often coupled with speech impediments.

Callers with Cognitive Impairments
This is even more vast than all the others combined but the most important consideration is to keep the jargon simple, particularly with what callers need to say back to the system. As an example, it was observed while listening to recorded calls from a pharmacy IVR that a caller had a hard time saying “My Prescriptions”. Overall it's always good to design with simplified wording in mind. And in the end, this holds true and is helpful for those callers without any impairments.