ECCH: Erasure Coded Consistent Hashing for Distributed Storage Systems

Yinglong Xiong, Jiang Zhou, Lin Su, Weiping Wang, Yong Chen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In this paper, we propose ECCH, an Erasure Coded Consistent Hashing scheme to make better data placement in distributed storage systems. It combines the inherent advantages of consistent hashing together with the storage-efficiency of erasure coding technology. Specifically, ECCH divides data block stream of files into groups according to block IDs. In each group, it encoded data blocks with additional parity blocks by erasure coding. All encoded blocks in the same group are stored on different nodes with consistent hashing distribution. For node failure or data loss, ECCH locates required data through the ID of missing block in a same group for fast recovery. To deal with node changes, ECCH introduces a design of multi-version hash rings to manage data layout. It can prevent the impact of data migration on erasure coding, while achieving data balance with little data movement. We have implemented ECCH on the Sheepdog, a distributed object-based storage system. Evaluation results show that ECCH can greatly improve the space utilization of hashing-based storage systems, while achieving efficient fault tolerance.

Original languageEnglish
Title of host publication19th IEEE International Symposium on Parallel and Distributed Processing with Applications, 11th IEEE International Conference on Big Data and Cloud Computing, 14th IEEE International Conference on Social Computing and Networking and 11th IEEE International Conference on Sustainable Computing and Communications, ISPA/BDCloud/SocialCom/SustainCom 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages177-184
Number of pages8
ISBN (Electronic)9781665435741
DOIs
StatePublished - 2021
Event19th IEEE International Symposium on Parallel and Distributed Processing with Applications, 11th IEEE International Conference on Big Data and Cloud Computing, 14th IEEE International Conference on Social Computing and Networking and 11th IEEE International Conference on Sustainable Computing and Communications, ISPA/BDCloud/SocialCom/SustainCom 2021 - New York, United States
Duration: Sep 30 2021Oct 3 2021

Publication series

Name19th IEEE International Symposium on Parallel and Distributed Processing with Applications, 11th IEEE International Conference on Big Data and Cloud Computing, 14th IEEE International Conference on Social Computing and Networking and 11th IEEE International Conference on Sustainable Computing and Communications, ISPA/BDCloud/SocialCom/SustainCom 2021

Conference

Conference19th IEEE International Symposium on Parallel and Distributed Processing with Applications, 11th IEEE International Conference on Big Data and Cloud Computing, 14th IEEE International Conference on Social Computing and Networking and 11th IEEE International Conference on Sustainable Computing and Communications, ISPA/BDCloud/SocialCom/SustainCom 2021
Country/TerritoryUnited States
CityNew York
Period09/30/2110/3/21

Keywords

  • Consistent hashing
  • Data distribution
  • Erasure coding
  • Fault tolerance
  • Replication

Fingerprint

Dive into the research topics of 'ECCH: Erasure Coded Consistent Hashing for Distributed Storage Systems'. Together they form a unique fingerprint.

Cite this