Collective input/output under memory constraints

Yin Lu, Yong Chen, Yu Zhuang, Jialin Liu, Rajeev Thakur

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

Compared with current high-performance computing (HPC) systems, exascale systems are expected to have much less memory per node, which can significantly reduce necessary collective input/output (I/O) performance. In this study, we introduce a memory-conscious collective I/O strategy that takes into account memory capacity and bandwidth constraints. The new strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intranode and internode layers, and determines I/O aggregators at run time considering memory consumption among processes. We have prototyped the design and evaluated it with commonly used benchmarks to verify its potential. The evaluation results demonstrate that this strategy holds promise in mitigating the memory pressure, alleviating the contention for memory bandwidth, and improving the I/O performance for projected extreme-scale systems. Given the importance of supporting increasingly data-intensive workloads and projected memory constraints on increasingly larger scale HPC systems, this new memory-conscious collective I/O can have a significant positive impact on scientific discovery productivity.

Original languageEnglish
Pages (from-to)21-36
Number of pages16
JournalInternational Journal of High Performance Computing Applications
Volume29
Issue number1
DOIs
StatePublished - Feb 13 2015

Keywords

  • Exascale system
  • collective input/output
  • data-intensive computing
  • high-performance computing
  • many-core architecture
  • parallel input/output

Fingerprint Dive into the research topics of 'Collective input/output under memory constraints'. Together they form a unique fingerprint.

Cite this