Lightweight Provenance Service for High-Performance Computing

Dong Dai, Yong Chen, Philip Carns, John Jenkins, Robert Ross

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data management needs.

Original languageEnglish
Title of host publicationProceedings - 26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages117-129
Number of pages13
ISBN (Electronic)9781467395243
DOIs
StatePublished - Oct 31 2017
Event26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017 - Portland, United States
Duration: Sep 9 2017Sep 13 2017

Publication series

NameParallel Architectures and Compilation Techniques - Conference Proceedings, PACT
Volume2017-September
ISSN (Print)1089-795X

Conference

Conference26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017
CountryUnited States
CityPortland
Period09/9/1709/13/17

Keywords

  • Data management
  • High-performance computing
  • Lightweight
  • Provenance
  • Rich metadata

Fingerprint Dive into the research topics of 'Lightweight Provenance Service for High-Performance Computing'. Together they form a unique fingerprint.

  • Cite this

    Dai, D., Chen, Y., Carns, P., Jenkins, J., & Ross, R. (2017). Lightweight Provenance Service for High-Performance Computing. In Proceedings - 26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017 (pp. 117-129). (Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT; Vol. 2017-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/PACT.2017.14