Memory-conscious collective I/O for extreme-scale HPC systems

The continuing decrease in memory capacity per core and the increasing disparity between core count and off-chip memory bandwidth create significant challenges for I/O operations in exascale systems. The exascale challenges require rethinking collective I/O for the effective exploitation of the correlation among I/O accesses in the exascale system. In this study, considering the major constraint of the memory space, we introduce a Memory-Conscious collective I/O. Given the importance of I/O aggregator in improving the performance of collective I/O, the new collective I/O strategy restricts aggregation data traffic within disjointed subgroups, coordinates I/O accesses in intra-node and inter-node layer and determines I/O aggregators at run time considering data distribution and memory consumption among processes. The preliminary results have demonstrated that the new collective I/O strategy holds promise in substantially reducing the amount of memory pressure, alleviating contention for memory bandwidth and improving the I/O performance for extreme-scale systems.