Memory-conscious collective I/O for extreme-scale HPC systems

Yin Lu, Yong Chen, Rajeev Thakur, Yu Zhuang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

The continuing decrease in memory capacity per core and the increasing disparity between core count and off-chip memory bandwidth create significant challenges for I/O operations in exascale systems. The exascale challenges require rethinking collective I/O for the effective exploitation of the correlation among I/O accesses in the exascale system. In this study, considering the major constraint of the memory space, we introduce a Memory-Conscious collective I/O. Given the importance of I/O aggregator in improving the performance of collective I/O, the new collective I/O strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intra-node and inter-node layer and determines I/O aggregators at run time considering data distribution and memory consumption among processes. The preliminary results have demonstrated that the new collective I/O strategy holds promise in substantially reducing the amount of memory pressure, alleviating contention for memory bandwidth and improving the I/O performance for extreme-scale systems.

Original languageEnglish
Title of host publicationProceedings - 2012 SC Companion
Subtitle of host publicationHigh Performance Computing, Networking Storage and Analysis, SCC 2012
Pages1360-1362
Number of pages3
DOIs
StatePublished - 2012
Event2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 - Salt Lake City, UT, United States
Duration: Nov 10 2012Nov 16 2012

Publication series

NameProceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Conference

Conference2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012
CountryUnited States
CitySalt Lake City, UT
Period11/10/1211/16/12

Fingerprint Dive into the research topics of 'Memory-conscious collective I/O for extreme-scale HPC systems'. Together they form a unique fingerprint.

  • Cite this

    Lu, Y., Chen, Y., Thakur, R., & Zhuang, Y. (2012). Memory-conscious collective I/O for extreme-scale HPC systems. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 (pp. 1360-1362). [6495972] (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.189