Efficient disk-to-disk sorting: A case study in the Decoupled Execution Paradigm

Hassan Eslami, Anthony Kougkas, Maria Kotsifakou, Theodoros Kasampalis, Kun Feng, Yin Lu, William Gropp, Xian He Sun, Yong Chen, Rajeev Thakur

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Many applications foreseen for exascale era should process huge amount of data. However, the IO infrastructure of current supercomputing architecture cannot be generalized to deal with this amount of data due to the need for excessive data movement from storage layers to compute nodes leading to limited scalability. There has been extensive studies addressing this challenge. Decoupled Execution Paradigm (DEP) is an attractive solution due to its unique features such as available fast storage devices close to computational units and available programmable units close to file system. In this paper we study the effectiveness of DEP for a well-known data-intensive kernel, disk-to-disk (aka out-of-core) sorting. We propose an optimized algorithm that uses almost all features of DEP pushing the performance of sorting in HPC even further compared to other existing solutions. Advantages in our algorithm are gained by exploiting programming units close to parallel file system to achieve higher IO throughput, compressing data before sending it over network or to disk, storing intermediate results of computation close to compute nodes, and fully overlapping IO with computation. We also provide an analytical model for our proposed algorithm. Our algorithm achieves 30% better performance compared to the theoretically optimal sorting algorithm running on the same testbed but not designed to exploit the DEP architecture.

Original languageEnglish
Title of host publicationProceedings of DISCS 2015
Subtitle of host publicationThe 2015 International Workshop on Data-Intensive Scalable Computing Systems - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450339933
DOIs
StatePublished - Nov 15 2015
EventInternational Workshop on Data-Intensive Scalable Computing Systems, DISCS 2015 - Austin, United States
Duration: Nov 15 2015 → …

Publication series

NameProceedings of DISCS 2015: The 2015 International Workshop on Data-Intensive Scalable Computing Systems - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

ConferenceInternational Workshop on Data-Intensive Scalable Computing Systems, DISCS 2015
CountryUnited States
CityAustin
Period11/15/15 → …

Keywords

  • Decoupled execution paradigm
  • Disk-to-disk sorting
  • Parallel IO
  • Parallel file system
  • Performance optimization

Fingerprint Dive into the research topics of 'Efficient disk-to-disk sorting: A case study in the Decoupled Execution Paradigm'. Together they form a unique fingerprint.

  • Cite this

    Eslami, H., Kougkas, A., Kotsifakou, M., Kasampalis, T., Feng, K., Lu, Y., Gropp, W., Sun, X. H., Chen, Y., & Thakur, R. (2015). Efficient disk-to-disk sorting: A case study in the Decoupled Execution Paradigm. In Proceedings of DISCS 2015: The 2015 International Workshop on Data-Intensive Scalable Computing Systems - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis [2] (Proceedings of DISCS 2015: The 2015 International Workshop on Data-Intensive Scalable Computing Systems - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/2831244.2831249