Interference-Aware I/O scheduling for data-intensive applications on hierarchical HPC storage systems

Scientific applications in critical areas are becoming more and more data-intensive. As data volume continues to grow, the data movement from memory to storage system has turned into a crucial performance bottleneck for many data-intensive applications. Newly emerged burst buffer concept provides a promising solution by increasing the depth of storage hierarchy to increase I/O performance of data-intensive applications. However, the data management on such multi-layer hierarchical storage system is still understudied. How to leverage each layer of storage for efficient data movement is an important research topic for HPC field. In this paper, we present a dynamic, interference-aware scheduling scheme that can efficiently manages the I/O scheduling among different layers of hierarchical HPC storage system to coordinate multiple concurrent data-intensive applications. Extensive experiments have been conducted and the results have demonstrated that our proposed approach can significantly improve the I/O performance of data-intensive applications.