Exploring Metadata Search Essentials for Scientific Data Management

Wei Zhang, Suren Byna, Chenxu Niu, Yong Chen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Scientific experiments and observations store massive amounts of data in various scientific file formats. Metadata, which describes the characteristics of the data, is commonly used to sift through massive datasets in order to locate data of interest to scientists. Several indexing data structures (such as hash tables, trie, self-balancing search trees, sparse array, etc.) have been developed as part of efforts to provide an efficient method for locating target data. However, efficient determination of an indexing data structure remains unclear in the context of scientific data management, due to the lack of investigation on metadata, metadata queries, and corresponding data structures. In this study, we perform a systematic study of the metadata search essentials in the context of scientific data management. We study a real-world astronomy observation dataset and explore the characteristics of the metadata in the dataset. We also study possible metadata queries based on the discovery of the metadata characteristics and evaluate different data structures for various types of metadata attributes. Our evaluation on real-world dataset suggests that trie is a suitable data structure when prefix/suffix query is required, otherwise hash table should be used. We conclude our study with a summary of our findings. These findings provide a guideline and offers insights in developing metadata indexing methodologies for scientific applications.

Original languageEnglish
Title of host publicationProceedings - 26th IEEE International Conference on High Performance Computing, HiPC 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages83-92
Number of pages10
ISBN (Electronic)9781728145358
DOIs
StatePublished - Dec 2019
Event26th Annual IEEE International Conference on High Performance Computing, HiPC 2019 - Hyderabad, India
Duration: Dec 17 2019Dec 20 2019

Publication series

NameProceedings - 26th IEEE International Conference on High Performance Computing, HiPC 2019

Conference

Conference26th Annual IEEE International Conference on High Performance Computing, HiPC 2019
CountryIndia
CityHyderabad
Period12/17/1912/20/19

Keywords

  • Data Management
  • HDF5
  • Metadata Indexing
  • Metadata Search

Fingerprint Dive into the research topics of 'Exploring Metadata Search Essentials for Scientific Data Management'. Together they form a unique fingerprint.

Cite this