With the continued improvement and innovation, technology has become an integral part of our daily lives. The rapid adoption of technology and its affordability has given rise to the Internet-of-Things (IoT). IoT is an interconnected network of devices that are able to communicate and share information seamlessly. IoT encompasses a gamut of heterogeneous devices ranging from a small sensor to large industrial machines. One such domain of IoT that has seen a significant growth in the recent few years is that of the wearable devices. While the privacy issues for medical devices has been well-researched and documented in the literature, the threats to privacy arising from the use of consumer wearable devices have received very little attention from the research community. This paper presents a survey of the literature to understand the various privacy challenges, mitigation strategies, and future research directions as a result of the widespread adoption of wearable devices.