Lip and Jaw Movements for Vowels and Consonants: Spatio-Temporal Characteristics and Bimodal Recognition Applications

This research focuses on the spatio-temporal characteristics of lips and
jaw movements and on their relevance for lip-reading, bimodal communication
theory and bimodal recognition applications. 3D visible articulatory targets for
vowels and consonants are proposed. Relevant modifications on the spatiotemporal
consonant targets due to coarticulatory phenomena are exemplified.
When visual parameters are added to acoustic ones as inputs to a Recurrent Neural
Network system, high recognition results in plosive classification experiments are
obtained.