Abstract
Distributed streaming dataflow systems have evolved into scalable and fault-tolerant production-grade systems. Their applicability has departed from the mere analysis of streaming windows and complex-event processing, and now includes cloud applications and machine learning inference. Although the advancements in the state management of streaming systems have contributed significantly to their maturity, the internal state of streaming operators has been so far hidden from external applications. However, that internal state can be seen as a materialized view that can be used for analytics, monitoring, and debugging. In this paper we argue that exposing the internal state of streaming systems to outside applications by making it queryable, opens the road for novel use cases. To this end, we introduce S-QUERY: an approach and reference architecture where the state of stream processors can be queried - either live or through snapshots, achieving different isolation levels. We show how this new capability can be implemented in an existing open-source stream processor, and how queryable state can affect the performance of such a system. Our experimental evaluation suggests that the snapshot configuration adds only up to 8ms latency in the 99.99thpercentile and negligible increase in 0-90thpercentiles.
Original language | English |
---|---|
Title of host publication | Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE) |
Editors | L. O'Conner |
Place of Publication | Piscataway |
Publisher | IEEE |
Pages | 1314-1327 |
Number of pages | 14 |
ISBN (Electronic) | 978-1-6654-0883-7 |
ISBN (Print) | 978-1-6654-0884-4 |
DOIs | |
Publication status | Published - 2022 |
Event | 2022 IEEE 38th International Conference on Data Engineering (ICDE) - Kuala Lumpur, Malaysia Duration: 9 May 2022 → 12 May 2022 |
Publication series
Name | Proceedings - International Conference on Data Engineering |
---|---|
Volume | 2022-May |
ISSN (Print) | 1084-4627 |
Conference
Conference | 2022 IEEE 38th International Conference on Data Engineering (ICDE) |
---|---|
Country/Territory | Malaysia |
City | Kuala Lumpur |
Period | 9/05/22 → 12/05/22 |
Bibliographical note
Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-careOtherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.