TY - GEN
T1 - Attributed consistent hashing for heterogeneous storage systems
AU - Zhou, Jiang
AU - Chen, Yong
AU - Wang, Weiping
N1 - Funding Information:
This research is supported in part by the National Science Foundation under grant CNS-1338078, CNS-1526055, IIP-1362134, CCF-1409946, and CCF-1718336. This research is also supported by the Beijing Municipal Science and Technology Project under grant Z181100002718004.
Publisher Copyright:
© 2018 Association for Computing Machinery.
PY - 2018/11/1
Y1 - 2018/11/1
N2 - Storage systems are critical building blocks of high-end computing systems and data centers. They demand the flexibility to distribute data effectively and provide high I/O performance. The consistent hashing algorithm is widely used in parallel/distributed file systems due to its decentralized design, scalability, and adaptability to node changes. However, it lacks efficiency in a heterogeneous environment where different storage devices, e.g. hard disk drives and solid state drives, co-exist. In this study, we propose an attributed consistent hashing (attributedCH), to overcome this deficiency. AttributedCH manages heterogeneous nodes on a consistent hashing ring and maintains attributes for each node to characterize distinct node features. It divides the hash ring into sectors and selects nodes from the sector with a comprehensive selection strategy. By considering different attributes, attributedCH achieves adaptive and efficient data placement for heterogeneous storage systems. We have carried out extensive evaluations and the evaluation results confirm that the attributedCH overcomes the deficiency of existing consistent hashing algorithms well and is particularly suitable for heterogeneous storage systems.
AB - Storage systems are critical building blocks of high-end computing systems and data centers. They demand the flexibility to distribute data effectively and provide high I/O performance. The consistent hashing algorithm is widely used in parallel/distributed file systems due to its decentralized design, scalability, and adaptability to node changes. However, it lacks efficiency in a heterogeneous environment where different storage devices, e.g. hard disk drives and solid state drives, co-exist. In this study, we propose an attributed consistent hashing (attributedCH), to overcome this deficiency. AttributedCH manages heterogeneous nodes on a consistent hashing ring and maintains attributes for each node to characterize distinct node features. It divides the hash ring into sectors and selects nodes from the sector with a comprehensive selection strategy. By considering different attributes, attributedCH achieves adaptive and efficient data placement for heterogeneous storage systems. We have carried out extensive evaluations and the evaluation results confirm that the attributedCH overcomes the deficiency of existing consistent hashing algorithms well and is particularly suitable for heterogeneous storage systems.
KW - Consistent hashing
KW - Data distribution
KW - Heterogeneous storage system
KW - Parallel file system
UR - http://www.scopus.com/inward/record.url?scp=85061548384&partnerID=8YFLogxK
U2 - 10.1145/3243176.3243202
DO - 10.1145/3243176.3243202
M3 - Conference contribution
AN - SCOPUS:85061548384
T3 - Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT
BT - Proceedings - 27th International Conference on Parallel Architectures and Compilation Techniques, PACT 2018
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 1 November 2018 through 4 November 2018
ER -