Multisensory Integration of Visual and Auditory Signals during Second Language Learning

Guangsheng Liang, Vinh Thai Nguyen, Kimi Nakatsukasa, Aaron Braver, Tommy Dang, Miranda Scolari

Research output: Other contributionpeer-review


Extensive research indicates that visual cues are integrated with auditory information to influence speed perception (e.g., the McGurk effect). In a separate line of work, researchers have recently reported that hand gestures are an effective tool to enhance auditory perception during second-language (L2) learning in natural classroom environments. This is particularly true in languages like Japanese, where long and short vowel durations convey meaning in a way that does not exist in English. Here, we investigate whether multisensory integration is the mechanism behind this phenomenon. To remove all co-occurring cues in a natural teaching environment other than the visual gestures, we designed a digital avatar instructor to deliver a 40-minute computer-assisted language training session. Twenty-nine native English speakers enrolled in the beginning or intermediate Japanese were randomly assigned to receive instruction from either a gesturing or stationary avatar, where the inclusion o
Original languageEnglish
PublisherThe Association for Research in Vision and Ophthalmology
StatePublished - Sep 1 2019


Dive into the research topics of 'Multisensory Integration of Visual and Auditory Signals during Second Language Learning'. Together they form a unique fingerprint.

Cite this