Elastic Consistent Hashing for Distributed Storage Systems

Wei Xie, Yong Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations

Abstract

Elastic distributed storage systems have been increasingly studied in recent years because power consumption has become a major problem in data centers. Much progress has been made in improving the agility of resizing small-A nd large-scale distributed storage systems. However, most of these studies focus on metadata based distributed storage systems. On the other hand, emerging consistent hashing based distributed storage systems are considered to allow better scalability and are highly attractive. We identify challenges in achieving elasticity in consistent hashing based distributed storage. These challenges cannot be easily solved by techniques used in current studies. In this paper, we propose an elastic consistent hashing based distributed storage to solve two problems. First, in order to allow a distributed storage to resize quickly, we modify the data placement algorithm using a primary server design and achieve an equal-work data layout. Second, we propose a selective data re-integration technique to reduce the performance impact when resizing a cluster. Our experimental and trace analysis results confirm that our proposed elastic consistent hashing works effectively and allows significantly better elasticity.

Original languageEnglish
Title of host publicationProceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages876-885
Number of pages10
ISBN (Electronic)9781538639146
DOIs
StatePublished - Jun 30 2017
Event31st IEEE International Parallel and Distributed Processing Symposium, IPDPS 2017 - Orlando, United States
Duration: May 29 2017Jun 2 2017

Publication series

NameProceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017

Conference

Conference31st IEEE International Parallel and Distributed Processing Symposium, IPDPS 2017
CountryUnited States
CityOrlando
Period05/29/1706/2/17

    Fingerprint

Cite this

Xie, W., & Chen, Y. (2017). Elastic Consistent Hashing for Distributed Storage Systems. In Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017 (pp. 876-885). [7967178] (Proceedings - 2017 IEEE 31st International Parallel and Distributed Processing Symposium, IPDPS 2017). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2017.88