TY - JOUR
T1 - Trigger-based Incremental Data Processing with Unified Sync and Async Model
AU - Dai, Dong
AU - Chen, Yong
AU - Kimpe, Dries
AU - Ross, Rob
N1 - Publisher Copyright:
IEEE
Copyright:
Copyright 2018 Elsevier B.V., All rights reserved.
PY - 2018/6/20
Y1 - 2018/6/20
N2 - In recent years, more and more applications in the cloud have needs to process large-scale on-line datasets, which evolve over time as new entries are added and existing entries are modified. Several programming frameworks, such as Percolator and Oolong, are proposed for such incremental data processing and can achieve efficient processing with an event-driven abstraction. However, these frameworks are inherently asynchronous, leaving the heavy burden of managing synchronization to applications' developers, which further significantly restricts their usability. In this study, we propose a trigger-based incremental computing framework for big data applications in the cloud, called Domino, with both synchronous and asynchronous mechanism to coordinate parallel triggers. With this new framework, both synchronous and asynchronous applications can be seamlessly developed. Use cases and extensive evaluation results confirm that it can deliver sufficient performance, and also is easy to use for incremental applications in large-scale distributed computing.
AB - In recent years, more and more applications in the cloud have needs to process large-scale on-line datasets, which evolve over time as new entries are added and existing entries are modified. Several programming frameworks, such as Percolator and Oolong, are proposed for such incremental data processing and can achieve efficient processing with an event-driven abstraction. However, these frameworks are inherently asynchronous, leaving the heavy burden of managing synchronization to applications' developers, which further significantly restricts their usability. In this study, we propose a trigger-based incremental computing framework for big data applications in the cloud, called Domino, with both synchronous and asynchronous mechanism to coordinate parallel triggers. With this new framework, both synchronous and asynchronous applications can be seamlessly developed. Use cases and extensive evaluation results confirm that it can deliver sufficient performance, and also is easy to use for incremental applications in large-scale distributed computing.
KW - Cloud
KW - Incremental Computing
KW - Programming Framework
UR - http://www.scopus.com/inward/record.url?scp=85048900341&partnerID=8YFLogxK
U2 - 10.1109/TCC.2018.2830348
DO - 10.1109/TCC.2018.2830348
M3 - Article
AN - SCOPUS:85048900341
JO - IEEE Transactions on Cloud Computing
JF - IEEE Transactions on Cloud Computing
SN - 2168-7161
ER -