Smartphone speech privacy concerns from side-channel attacks on facial biomechanics

Speech is a complex orchestration of physical movements which involves the lungs, vocal cords, face, jaw, and mouth. As we speak on the phone, we inadvertently impart energy on the mobile device at our ear, causing it to move as our face shapes words and sounds. We theorize that different phonetics from the International Phonetic Alphabet (IPA), which act as the building blocks of speech, may have their own fingerprint on motion sensor data during a phone conversation. When phonetics are combined into words, the relationship between phonetics and motion sensor data could cause words to also be identifiable. Based on an initial investigation into the relationship between phonetics and motion sensor data, we develop attacks to evaluate the risk that this could pose to user privacy. We evaluate attacks for classifying digits, differentiating between digit and non-digit speech, identifying the gender of the user, and user identification. The results of these experiments in various configurations demonstrate that the attacks can be highly effective. Our research adds to the body of work making the case for additional measures to control and protect data produced by users and their devices. Without action on the part of technology producers, users will remain vulnerable to attacks which leverage APIs that leave the user without any ability to control the data that their devices generate.