GraphMeta: A graph-based engine for managing large-scale HPC rich metadata

Dong Dai, Yong Chen, Philip Carns, John Jenkins, Wei Zhang, Robert Ross

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Scopus citations

Abstract

High-performance computing (HPC) systems face increasingly critical metadata management challenges, especially in the approaching exascale era. These challenges arise not only from exploding metadata volumes but also from increasingly diverse metadata, which contains data provenance and userdefined attributes in addition to traditional POSIX metadata. This "rich" metadata is critical to support many advanced data management functionality such as data auditing and validation. In our prior work, we presented a graph-based model that could be a promising solution to uniformly manage such rich metadata because of its flexibility and generality. At the same time, however, graph-based rich metadata management introduces significant challenges. In this study, we first identify the challenges presented by the underlying infrastructure in supporting scalable, high-performance rich metadata management. To tackle these challenges, we then present GraphMeta, a graph-based engine designed for managing large-scale rich metadata. We also utilize a series of optimizations designed for rich metadata graphs. We evaluate GraphMeta with both synthetic and real HPC metadata workloads and compare it with other approaches. The results show that its advantages in terms of rich metadata management in HPC systems, including better performance and scalability compared with existing solutions.

Original languageEnglish
Title of host publicationProceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages298-307
Number of pages10
ISBN (Electronic)9781509036530
DOIs
StatePublished - Dec 6 2016
Event2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 - Taipei, Taiwan, Province of China
Duration: Sep 13 2016Sep 15 2016

Publication series

NameProceedings - IEEE International Conference on Cluster Computing, ICCC
ISSN (Print)1552-5244

Conference

Conference2016 IEEE International Conference on Cluster Computing, CLUSTER 2016
CountryTaiwan, Province of China
CityTaipei
Period09/13/1609/15/16

Fingerprint Dive into the research topics of 'GraphMeta: A graph-based engine for managing large-scale HPC rich metadata'. Together they form a unique fingerprint.

  • Cite this

    Dai, D., Chen, Y., Carns, P., Jenkins, J., Zhang, W., & Ross, R. (2016). GraphMeta: A graph-based engine for managing large-scale HPC rich metadata. In Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 (pp. 298-307). [7776522] (Proceedings - IEEE International Conference on Cluster Computing, ICCC). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTER.2016.50