A Holistic Heterogeneity-Aware Data Placement Scheme for Hybrid Parallel I/O Systems

Shuibing He, Zheng Li, Jiang Zhou, Yanlong Yin, Xiaohua Xu, Yong Chen, Xian He Sun

Research output: Contribution to journalArticlepeer-review

Abstract

We present H2DP, a holistic heterogeneity-aware data placement scheme for hybrid parallel I/O systems, which consist of HDD servers and SSD servers. Most of the existing approaches focus on server performance or application I/O pattern heterogeneity in data placement. H2DP considers three axes of heterogeneity: server performance, server space, and application I/O pattern. More specifically, H2DP determines the optimized stripe sizes on servers based on server performance, keeps only critical data on all hybrid servers and the rest data on HDD servers, and dynamically migrates data among different types of servers at run-time. This holistic heterogeneity-awareness enables H2DP to achieve high performance by alleviating server load imbalance, efficiently utilizing SSD space, and accommodating application pattern variation. We have implemented a prototype of H2DP under MPICH2 atop OrangeFS. Extensive experimental results demonstrate that H2DP significantly improve I/O system performance compared to existing data placement schemes.

Original languageEnglish
Article number8880508
Pages (from-to)830-842
Number of pages13
JournalIEEE Transactions on Parallel and Distributed Systems
Volume31
Issue number4
DOIs
StatePublished - Apr 1 2020

Keywords

  • Parallel I/O system
  • data placement
  • hybrid parallel file system
  • parallel file system
  • solid state drive

Fingerprint

Dive into the research topics of 'A Holistic Heterogeneity-Aware Data Placement Scheme for Hybrid Parallel I/O Systems'. Together they form a unique fingerprint.

Cite this