Active burst-buffer: In-transit processing integrated into hierarchical storage

Chao Chen, Michael Lang, Latchesar Ionkov, Yong Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

The data volume of many scientific applications has substantially increased in the past decade and continues to increase due to the rising needs of high-resolution and fine- granularity scientific discovery. The data movement between stor- age and compute nodes has become a critical performance factor and has attracted intense research and development attention in recent years. In this paper, we propose a novel solution, named Active burst-buffer, to reduce the unnecessary data movement and to speed up scientific workflow. Active burst-buffer enhances the existing burst-buffer concept with analysis capabilities by reconstructing the cached data to a logic file and providing a MapReduce-like computing framework for programming and executing the analysis codes. An extensive set of experiments were conducted to evaluate the performance of Active burst-buffer by comparing it against existing mainstream schemes, and more than 30% improvements were observed. The evaluations confirm that Active burst-buffer is capable of enabling efficient data analysis in-transit on burst-buffer nodes and is a promising solution to scientific discoveries with large-scale data sets.

Original languageEnglish
Title of host publication2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509033157
DOIs
StatePublished - Aug 23 2016
Event11th IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Long Beach, United States
Duration: Aug 8 2016Aug 10 2016

Publication series

Name2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings

Conference

Conference11th IEEE International Conference on Networking Architecture and Storage, NAS 2016
CountryUnited States
CityLong Beach
Period08/8/1608/10/16

Fingerprint Dive into the research topics of 'Active burst-buffer: In-transit processing integrated into hierarchical storage'. Together they form a unique fingerprint.

  • Cite this

    Chen, C., Lang, M., Ionkov, L., & Chen, Y. (2016). Active burst-buffer: In-transit processing integrated into hierarchical storage. In 2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings [7549390] (2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/NAS.2016.7549390