Hierarchical I/O scheduling for collective I/O

Jialin Liu, Yong Chen, Yu Zhuang

Research output: Contribution to conferencePaperpeer-review

16 Scopus citations


The non-contiguous access pattern of many scientific applications results in a large number of I/O requests, which can seriously limit the data-access performance. Collective I/O has been widely used to address this issue. However, the performance of collective I/O could be dramatically degraded in today's high-performance computing system due to the increasing shuffle cost caused by highly concurrent data accesses. This situation tends to be even worse as many applications become more and more data intensive. Previous research has primarily focused on optimizing I/O access cost in collective I/O but largely ignored the shuffle cost involved. In this study, we propose a new hierarchical I/O scheduling (HIO) algorithm to address the increasing shuffle cost in collective I/O. The fundamental idea is to schedule applications' I/O requests based on a shuffle cost analysis to achieve the optimal overall performance, instead of achieving optimal I/O accesses only. The algorithm is currently evaluated with the MPICH2 and PVFS2. Both theoretical analysis and experimental tests show that the proposed hierarchical I/O scheduling has a potential in addressing the degraded performance issue of collective I/O with highly concurrent accesses.

Original languageEnglish
Number of pages8
StatePublished - 2013
Event13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013 - Delft, Netherlands
Duration: May 13 2013May 16 2013


Conference13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013


  • Big data
  • Collective I/O
  • Data intensive computing
  • High-performance computing
  • Scheduling


Dive into the research topics of 'Hierarchical I/O scheduling for collective I/O'. Together they form a unique fingerprint.

Cite this