Interference-Aware I/O scheduling for data-intensive applications on hierarchical HPC storage systems

Weihao Liang, Yong Chen, Hong An

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Scientific applications in critical areas are becoming more and more data-intensive. As data volume continues to grow, the data movement from memory to storage system has turned into a crucial performance bottleneck for many data-intensive applications. Newly emerged burst buffer concept provides a promising solution by increasing the depth of storage hierarchy to increase I/O performance of data-intensive applications. However, the data management on such multi-layer hierarchical storage system is still understudied. How to leverage each layer of storage for efficient data movement is an important research topic for HPC field. In this paper, we present a dynamic, interference-aware scheduling scheme that can efficiently manages the I/O scheduling among different layers of hierarchical HPC storage system to coordinate multiple concurrent data-intensive applications. Extensive experiments have been conducted and the results have demonstrated that our proposed approach can significantly improve the I/O performance of data-intensive applications.

Original languageEnglish
Title of host publicationProceedings - 21st IEEE International Conference on High Performance Computing and Communications, 17th IEEE International Conference on Smart City and 5th IEEE International Conference on Data Science and Systems, HPCC/SmartCity/DSS 2019
EditorsZheng Xiao, Laurence T. Yang, Pavan Balaji, Tao Li, Keqin Li, Albert Zomaya
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages654-661
Number of pages8
ISBN (Electronic)9781728120584
DOIs
StatePublished - Aug 2019
Event21st IEEE International Conference on High Performance Computing and Communications, 17th IEEE International Conference on Smart City and 5th IEEE International Conference on Data Science and Systems, HPCC/SmartCity/DSS 2019 - Zhangjiajie, China
Duration: Aug 10 2019Aug 12 2019

Publication series

NameProceedings - 21st IEEE International Conference on High Performance Computing and Communications, 17th IEEE International Conference on Smart City and 5th IEEE International Conference on Data Science and Systems, HPCC/SmartCity/DSS 2019

Conference

Conference21st IEEE International Conference on High Performance Computing and Communications, 17th IEEE International Conference on Smart City and 5th IEEE International Conference on Data Science and Systems, HPCC/SmartCity/DSS 2019
CountryChina
CityZhangjiajie
Period08/10/1908/12/19

Keywords

  • Burst buffer
  • Data movement
  • I/O interference
  • I/O scheduling
  • Storage system

Fingerprint Dive into the research topics of 'Interference-Aware I/O scheduling for data-intensive applications on hierarchical HPC storage systems'. Together they form a unique fingerprint.

Cite this