Decoupled I/O for Data-Intensive High Performance Computing

Chao Chen, Yong Chen, Kun Feng, Yanlong Yin, Hassan Eslami, Rajeev Thakur, Xian He Sun, William D. Gropp

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

The I/O bottleneck issue has been acknowledged as one of main performance issues of high performance computing (HPC) systems for data-intensive scientific applications, and has attracted intensive studies in recent years. With the enlarging gap between the computing bandwidth and I/O bandwidth in projected next-generation HPC systems, this issue will become even worse. In this paper, we present a novel decoupledI/O to address the fundamental I/O bottleneck issue. The decoupled I/O is a software stack including MPI extensions, compiler improvements, and runtime library support, based one decoupled HPC system architecture. It allows users to treat the computing of data-intensive operations and the traditionalI/O operation as an ensemble and offload them into dedicateddata nodes, which are near to the data source, to reduce the overhead of data movement and improve the I/O bandwidth usage. The decoupled I/O is user-friendly and requires littlechanges in application codes. Experiments were conducted to evaluate the performance of the decoupled I/O, and the results show that it outperforms existing solutions (such as active storage I/O) and provides an attractive I/O solution for data-intensive high performance computing.

Original languageEnglish
Title of host publicationProceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages312-320
Number of pages9
ISBN (Electronic)9781479956159
DOIs
StatePublished - May 7 2015
Event43rd International Conference on Parallel Processing Workshops, ICPPW 2014 - Minneapolis, United States
Duration: Sep 9 2014Sep 12 2014

Publication series

NameProceedings of the International Conference on Parallel Processing Workshops
Volume2015-May
ISSN (Print)1530-2016

Conference

Conference43rd International Conference on Parallel Processing Workshops, ICPPW 2014
CountryUnited States
CityMinneapolis
Period09/9/1409/12/14

Keywords

  • Decoupled I/O
  • data-intensive computing
  • high performance computing
  • parallel I/O
  • storage

Fingerprint Dive into the research topics of 'Decoupled I/O for Data-Intensive High Performance Computing'. Together they form a unique fingerprint.

Cite this