MAMS: A highly reliable policy for metadata service

Jiang Zhou, Yong Chen, Weiping Wang, Dan Meng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Most mass data processing applications nowadays often need long, continuous, and uninterrupted data access. Parallel/distributed file systems often use multiple metadata servers to manage the global namespace and provide a reliability guarantee. With the rapid increase of data amount and system scale, the probability of hardware or software failures keeps increasing, which easily leads to multiple points of failures. Metadata service reliability has become a crucial issue as it affects file and directory operations in the event of failures. Existing reliable metadata management mechanisms can provide fault tolerance but have disadvantages in system availability, state consistence, and performance overhead. This paper introduces a new highly reliable policy called MAMS (multiple actives multiple standbys) to ensure multiple metadata service reliability in file systems. Different from traditional strategies, the MAMS divides metadata servers into different replica groups and maintains more than one standby node for failover in each group. Combining the global view with distributed protocols, the MAMS achieves an automatic state transition and service takeover. We have implemented the MAMS policy in a prototyping file system and conducted extensive tests to validate and evaluate it. The experimental results confirm that the MAMS policy can achieve a faster transparent fault tolerance in different error scenarios with less influence on metadata operations. Compared with typical designs in Hadoop Avatar, Hadoop HA, and Boom-FS file systems, the mean time to recovery (MTTR) with the MAMS was reduced by 80.23%, 65.46% and 28.13%, respectively.

Original languageEnglish
Title of host publicationProceedings - 2015 44th International Annual Conference on Parallel Processing, ICPP 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages729-738
Number of pages10
ISBN (Electronic)9781467375870
DOIs
StatePublished - Dec 8 2015
Event44th International Conference on Parallel Processing, ICPP 2015 - Beijing, China
Duration: Sep 1 2015Sep 4 2015

Publication series

NameProceedings of the International Conference on Parallel Processing
Volume2015-December
ISSN (Print)0190-3918

Conference

Conference44th International Conference on Parallel Processing, ICPP 2015
CountryChina
CityBeijing
Period09/1/1509/4/15

Keywords

  • Cluster file systems
  • Fault tolerance
  • Metadata management
  • Multiple metadata service
  • Parallel file systems

Fingerprint Dive into the research topics of 'MAMS: A highly reliable policy for metadata service'. Together they form a unique fingerprint.

  • Cite this

    Zhou, J., Chen, Y., Wang, W., & Meng, D. (2015). MAMS: A highly reliable policy for metadata service. In Proceedings - 2015 44th International Annual Conference on Parallel Processing, ICPP 2015 (pp. 729-738). [7349628] (Proceedings of the International Conference on Parallel Processing; Vol. 2015-December). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICPP.2015.82