Multisensory Integration of Visual and Auditory Signals during Second Language Learning

Extensive research indicates that visual cues are integrated with auditory information to influence speed perception (e.g., the McGurk effect). In a separate line of work, researchers have recently reported that hand gestures are an effective tool to enhance auditory perception during second-language (L2) learning in natural classroom environments. This is particularly true in languages like Japanese, where long and short vowel durations convey meaning in a way that does not exist in English. Here, we investigate whether multisensory integration is the mechanism behind this phenomenon. To remove all co-occurring cues in a natural teaching environment other than the visual gestures, we designed a digital avatar instructor to deliver a 40-minute computer-assisted language training session. Twenty-nine native English speakers enrolled in the beginning or intermediate Japanese were randomly assigned to receive instruction from either a gesturing or stationary avatar, where the inclusion o