TY - GEN
T1 - Lightweight Provenance Service for High-Performance Computing
AU - Dai, Dong
AU - Chen, Yong
AU - Carns, Philip
AU - Jenkins, John
AU - Ross, Robert
N1 - Publisher Copyright:
© 2017 IEEE.
PY - 2017/10/31
Y1 - 2017/10/31
N2 - Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data management needs.
AB - Provenance describes detailed information about the history of a piece of data, containing the relationships among elements such as users, processes, jobs, and workflows that contribute to the existence of data. Provenance is key to supporting many data management functionalities that are increasingly important in operations such as identifying data sources, parameters, or assumptions behind a given result; auditing data usage; or understanding details about how inputs are transformed into outputs. Despite its importance, however, provenance support is largely underdeveloped in highly parallel architectures and systems. One major challenge is the demanding requirements of providing provenance service in situ. The need to remain lightweight and to be always on often conflicts with the need to be transparent and offer an accurate catalog of details regarding the applications and systems. To tackle this challenge, we introduce a lightweight provenance service, called LPS, for high-performance computing (HPC) systems. LPS leverages a kernel instrument mechanism to achieve transparency and introduces representative execution and flexible granularity to capture comprehensive provenance with controllable overhead. Extensive evaluations and use cases have confirmed its efficiency and usability. We believe that LPS can be integrated into current and future HPC systems to support a variety of data management needs.
KW - Data management
KW - High-performance computing
KW - Lightweight
KW - Provenance
KW - Rich metadata
UR - http://www.scopus.com/inward/record.url?scp=85043572310&partnerID=8YFLogxK
U2 - 10.1109/PACT.2017.14
DO - 10.1109/PACT.2017.14
M3 - Conference contribution
AN - SCOPUS:85043572310
T3 - Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT
SP - 117
EP - 129
BT - Proceedings - 26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017
Y2 - 9 September 2017 through 13 September 2017
ER -