A hybrid shared-nothing/shared-data storage scheme for large-scale data processing

Huaiming Song, Xian He Sun, Yong Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Shared-nothing and shared-disk are the two most common storage architectures of parallel databases in the past two decades. Both two types of systems have their own merits for different applications. However, there are no much efforts in investigating the integration of these two architectures and exploiting their merits together. In this paper, we propose a novel hybrid storage architecture for large-scale data processing, to leverage the benefits of both shared-nothing and shared-disk architectures. In the proposed hybrid system, we adopt a shared-nothing architecture as the hardware layer and leverage a parallel file system as the storage layer to combine the scattered disks on all database nodes. We present an overall design of the new scheme, including data and storage organization, data access modes, and query processing methods. The proposed hybrid scheme can achieve both high I/O performance as a shared-nothing system, and high-speed data sharing across all server nodes as a share-disk system. Preliminary experimental results demonstrate that the hybrid scheme is promising and more appropriate for large-scale and data-intensive applications than each of the two individual types of systems.

Original languageEnglish
Title of host publicationProceedings - 9th IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2011
Pages161-166
Number of pages6
DOIs
StatePublished - 2011
Event9th IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2011 - Busan, Korea, Republic of
Duration: May 26 2011May 28 2011

Publication series

NameProceedings - 9th IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2011

Conference

Conference9th IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2011
CountryKorea, Republic of
CityBusan
Period05/26/1105/28/11

Keywords

  • Parallel databases
  • Parallel file systems
  • Shared-data
  • Shared-nothing
  • Storage architecture

Fingerprint Dive into the research topics of 'A hybrid shared-nothing/shared-data storage scheme for large-scale data processing'. Together they form a unique fingerprint.

  • Cite this

    Song, H., Sun, X. H., & Chen, Y. (2011). A hybrid shared-nothing/shared-data storage scheme for large-scale data processing. In Proceedings - 9th IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2011 (pp. 161-166). [5951899] (Proceedings - 9th IEEE International Symposium on Parallel and Distributed Processing with Applications, ISPA 2011). https://doi.org/10.1109/ISPA.2011.43