SUORA: A scalable and uniform data distribution algorithm for heterogeneous storage systems

Jiang Zhou, Wei Xie, Jason Noble, Kace Echo, Yong Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations

Abstract

The data scale in many data centers is growing explosively with emerging applications and usages of big data technologies. Data distribution is a key issue in large-scale distributed storage systems to place petabytes of data or even beyond, among tens or hundreds of thousands of storage devices. In the meantime, heterogeneous storage systems, such as those having devices with hard disk drives (HDDs) and storage class memories (SCMs), have become increasingly popular for massive data storage due to balanced performance, capacity, and cost. Current data distribution algorithms can achieve efficient, scalable, and balanced mapping, but do not distinguish different characteristics of heterogeneous devices well. This paper presents a novel data distribution algorithm called SUORA (Scalable and Uniform storage via Optimally-adaptive and Random number Addressing), to take full advantage of heterogeneous devices. SUORA is a pseudo-random algorithm that uniformly distributes data cross a hybrid and tiered storage cluster. It divides heterogeneous devices, maps them onto different buckets and assigns them to various segments in each bucket. A pseudo-random and deterministic number sequence is generated to map data among segments and devices. Data movement is performed for achieving better read throughput while keeping load balance according to data hotness and bucket threshold. With considering distinct characteristics of heterogeneous storage devices well, the SUORA algorithm achieves a highly efficient adaptive data distribution for data centers and heterogeneous storage systems.

Original languageEnglish
Title of host publication2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509033157
DOIs
StatePublished - Aug 23 2016
Event11th IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Long Beach, United States
Duration: Aug 8 2016Aug 10 2016

Publication series

Name2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings

Conference

Conference11th IEEE International Conference on Networking Architecture and Storage, NAS 2016
CountryUnited States
CityLong Beach
Period08/8/1608/10/16

Keywords

  • Data centers
  • Data distribution algorithm
  • Data management
  • Data placement
  • Heterogeneous storage

Fingerprint Dive into the research topics of 'SUORA: A scalable and uniform data distribution algorithm for heterogeneous storage systems'. Together they form a unique fingerprint.

Cite this