TY - GEN
T1 - Active burst-buffer
T2 - 11th IEEE International Conference on Networking Architecture and Storage, NAS 2016
AU - Chen, Chao
AU - Lang, Michael
AU - Ionkov, Latchesar
AU - Chen, Yong
N1 - Publisher Copyright:
© 2016 IEEE.
PY - 2016/8/23
Y1 - 2016/8/23
N2 - The data volume of many scientific applications has substantially increased in the past decade and continues to increase due to the rising needs of high-resolution and fine- granularity scientific discovery. The data movement between stor- age and compute nodes has become a critical performance factor and has attracted intense research and development attention in recent years. In this paper, we propose a novel solution, named Active burst-buffer, to reduce the unnecessary data movement and to speed up scientific workflow. Active burst-buffer enhances the existing burst-buffer concept with analysis capabilities by reconstructing the cached data to a logic file and providing a MapReduce-like computing framework for programming and executing the analysis codes. An extensive set of experiments were conducted to evaluate the performance of Active burst-buffer by comparing it against existing mainstream schemes, and more than 30% improvements were observed. The evaluations confirm that Active burst-buffer is capable of enabling efficient data analysis in-transit on burst-buffer nodes and is a promising solution to scientific discoveries with large-scale data sets.
AB - The data volume of many scientific applications has substantially increased in the past decade and continues to increase due to the rising needs of high-resolution and fine- granularity scientific discovery. The data movement between stor- age and compute nodes has become a critical performance factor and has attracted intense research and development attention in recent years. In this paper, we propose a novel solution, named Active burst-buffer, to reduce the unnecessary data movement and to speed up scientific workflow. Active burst-buffer enhances the existing burst-buffer concept with analysis capabilities by reconstructing the cached data to a logic file and providing a MapReduce-like computing framework for programming and executing the analysis codes. An extensive set of experiments were conducted to evaluate the performance of Active burst-buffer by comparing it against existing mainstream schemes, and more than 30% improvements were observed. The evaluations confirm that Active burst-buffer is capable of enabling efficient data analysis in-transit on burst-buffer nodes and is a promising solution to scientific discoveries with large-scale data sets.
UR - http://www.scopus.com/inward/record.url?scp=84988377989&partnerID=8YFLogxK
U2 - 10.1109/NAS.2016.7549390
DO - 10.1109/NAS.2016.7549390
M3 - Conference contribution
AN - SCOPUS:84988377989
T3 - 2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings
BT - 2016 IEEE International Conference on Networking Architecture and Storage, NAS 2016 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 8 August 2016 through 10 August 2016
ER -