Fast data analysis with integrated statistical metadata in scientific datasets

Jialin Liu; Yong Chen

doi:10.1109/ICPPW.2012.89

Fast data analysis with integrated statistical metadata in scientific datasets

Jialin Liu, Yong Chen

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Scientific datasets, such as HDF5 and PnetCDF, have been used widely in many scientific applications. These data formats and libraries provide essential support for data analysis in scientific discovery and innovations. In this research, we present an approach to boost data analysis, namely Fast Analysis with Statistical Metadata (FASM), via data sub setting and integrating a small amount of statistics into datasets. We discuss how the FASM can improve data analysis performance. It is currently evaluated with the PnetCDF on synthetic and real data, but can also be implemented in other libraries. The FASM can potentially lead to a new dataset design and can have an impact on data analysis.

Original language	English
Title of host publication	Proceedings - 41st International Conference on Parallel Processing Workshops, ICPPW 2012
Pages	602-603
Number of pages	2
DOIs	https://doi.org/10.1109/ICPPW.2012.89
State	Published - 2012
Event	41st International Conference on Parallel Processing Workshops, ICPPW 2012 - Pittsburgh, PA, United States Duration: Sep 10 2012 → Sep 13 2012

Publication series

Name	Proceedings of the International Conference on Parallel Processing Workshops
ISSN (Print)	1530-2016

Conference

Conference	41st International Conference on Parallel Processing Workshops, ICPPW 2012
Country/Territory	United States
City	Pittsburgh, PA
Period	09/10/12 → 09/13/12

Keywords

FASM
big data
data intensive computing
high performance computing
statistical techniques

Access to Document

10.1109/ICPPW.2012.89

Cite this

@inproceedings{9e9590b17dd64120814e44c4871bed19,

title = "Fast data analysis with integrated statistical metadata in scientific datasets",

abstract = "Scientific datasets, such as HDF5 and PnetCDF, have been used widely in many scientific applications. These data formats and libraries provide essential support for data analysis in scientific discovery and innovations. In this research, we present an approach to boost data analysis, namely Fast Analysis with Statistical Metadata (FASM), via data sub setting and integrating a small amount of statistics into datasets. We discuss how the FASM can improve data analysis performance. It is currently evaluated with the PnetCDF on synthetic and real data, but can also be implemented in other libraries. The FASM can potentially lead to a new dataset design and can have an impact on data analysis.",

keywords = "FASM, big data, data intensive computing, high performance computing, statistical techniques",

author = "Jialin Liu and Yong Chen",

year = "2012",

doi = "10.1109/ICPPW.2012.89",

language = "English",

isbn = "9780769547954",

series = "Proceedings of the International Conference on Parallel Processing Workshops",

pages = "602--603",

booktitle = "Proceedings - 41st International Conference on Parallel Processing Workshops, ICPPW 2012",

note = "41st International Conference on Parallel Processing Workshops, ICPPW 2012 ; Conference date: 10-09-2012 Through 13-09-2012",

}

Liu, J & Chen, Y 2012, Fast data analysis with integrated statistical metadata in scientific datasets. in Proceedings - 41st International Conference on Parallel Processing Workshops, ICPPW 2012., 6337537, Proceedings of the International Conference on Parallel Processing Workshops, pp. 602-603, 41st International Conference on Parallel Processing Workshops, ICPPW 2012, Pittsburgh, PA, United States, 09/10/12. https://doi.org/10.1109/ICPPW.2012.89

Fast data analysis with integrated statistical metadata in scientific datasets. / Liu, Jialin; Chen, Yong.
Proceedings - 41st International Conference on Parallel Processing Workshops, ICPPW 2012. 2012. p. 602-603 6337537 (Proceedings of the International Conference on Parallel Processing Workshops).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Fast data analysis with integrated statistical metadata in scientific datasets

AU - Liu, Jialin

AU - Chen, Yong

PY - 2012

Y1 - 2012

N2 - Scientific datasets, such as HDF5 and PnetCDF, have been used widely in many scientific applications. These data formats and libraries provide essential support for data analysis in scientific discovery and innovations. In this research, we present an approach to boost data analysis, namely Fast Analysis with Statistical Metadata (FASM), via data sub setting and integrating a small amount of statistics into datasets. We discuss how the FASM can improve data analysis performance. It is currently evaluated with the PnetCDF on synthetic and real data, but can also be implemented in other libraries. The FASM can potentially lead to a new dataset design and can have an impact on data analysis.

AB - Scientific datasets, such as HDF5 and PnetCDF, have been used widely in many scientific applications. These data formats and libraries provide essential support for data analysis in scientific discovery and innovations. In this research, we present an approach to boost data analysis, namely Fast Analysis with Statistical Metadata (FASM), via data sub setting and integrating a small amount of statistics into datasets. We discuss how the FASM can improve data analysis performance. It is currently evaluated with the PnetCDF on synthetic and real data, but can also be implemented in other libraries. The FASM can potentially lead to a new dataset design and can have an impact on data analysis.

KW - FASM

KW - big data

KW - data intensive computing

KW - high performance computing

KW - statistical techniques

UR - http://www.scopus.com/inward/record.url?scp=84871148080&partnerID=8YFLogxK

U2 - 10.1109/ICPPW.2012.89

DO - 10.1109/ICPPW.2012.89

M3 - Conference contribution

AN - SCOPUS:84871148080

SN - 9780769547954

T3 - Proceedings of the International Conference on Parallel Processing Workshops

SP - 602

EP - 603

BT - Proceedings - 41st International Conference on Parallel Processing Workshops, ICPPW 2012

T2 - 41st International Conference on Parallel Processing Workshops, ICPPW 2012

Y2 - 10 September 2012 through 13 September 2012

ER -

Fast data analysis with integrated statistical metadata in scientific datasets

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this