The Vocal Joystick Project seeks to develop a continuous and discrete control device based on continuous and discrete dimensions of human vocalizations. There are four continuous dimensions that can be extracted from vowel-like vocalizations: vowel height, vowel backness, pitch, and intensity. In addition to these, discrete control can be achieved through brief obstruent-like sounds and through words. The extracted dimensions can be used as control parameters for a variety of devices. The most obvious application is computer control in GUI environments, but applications also include robotic arms, thermostats, and lighting levels.
Bilmes, J., Malkin, J., Li, X., Harada, S., Kilanski, K., Kirchhoff, K., Wright, R.A., Subramanya, A., Landay, J., Dowden, P., & Chizeck, H. (2006). The Vocal Joystick. IEEE International Conference on Audio, Speech and Signal Processing, Toulouse, France, May 2006
Kilanski, K., Malkin, J., Li, X., Wright, R.A., & Bilmes, J. (2006). The Vocal Joystick Data Collection Effort and Vowel Corpus. Interspeech, Pittsburgh, Sep. 2006.
Li, X., Malkin, J., Harada, S., Bilmes, J., Wright, R.A., & Landay, J. (2006). An Online Adaptive Filtering Algorithm for the Vocal Joystick. Interspeech, Pittsburgh, Sep. 2006
Bilmes, J., Li, X., Malkin, J., Kilanski, K., Wright, R.A., Kirchhoff, K., Subramanya, A., Harada, S., Landay, J., Dowden, P., & Chizeck, H. (2005). The Vocal Joystick: A Voice-Based Human-Computer Interface for Individuals with Motor Impairments. Human Language Technology Conf. and Conf. on Empirical Methods in Natural Language Processing, Vancouver, Canada, Oct. 2005