Hierarchical I/O scheduling for collective I/O

Jialin Liu, Yong Chen, Yu Zhuang

Research output: Contribution to conferencePaperpeer-review

15 Scopus citations

Abstract

The non-contiguous access pattern of many scientific applications results in a large number of I/O requests, which can seriously limit the data-access performance. Collective I/O has been widely used to address this issue. However, the performance of collective I/O could be dramatically degraded in today's high-performance computing system due to the increasing shuffle cost caused by highly concurrent data accesses. This situation tends to be even worse as many applications become more and more data intensive. Previous research has primarily focused on optimizing I/O access cost in collective I/O but largely ignored the shuffle cost involved. In this study, we propose a new hierarchical I/O scheduling (HIO) algorithm to address the increasing shuffle cost in collective I/O. The fundamental idea is to schedule applications' I/O requests based on a shuffle cost analysis to achieve the optimal overall performance, instead of achieving optimal I/O accesses only. The algorithm is currently evaluated with the MPICH2 and PVFS2. Both theoretical analysis and experimental tests show that the proposed hierarchical I/O scheduling has a potential in addressing the degraded performance issue of collective I/O with highly concurrent accesses.

Original languageEnglish
Pages211-218
Number of pages8
DOIs
StatePublished - 2013
Event13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013 - Delft, Netherlands
Duration: May 13 2013May 16 2013

Conference

Conference13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013
Country/TerritoryNetherlands
CityDelft
Period05/13/1305/16/13

Keywords

  • Big data
  • Collective I/O
  • Data intensive computing
  • High-performance computing
  • Scheduling

Fingerprint

Dive into the research topics of 'Hierarchical I/O scheduling for collective I/O'. Together they form a unique fingerprint.

Cite this