Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems

Jonas Traub, Philipp Marian Grulich, Alejandro Rodríguez Cuéllar, Sebastian Breß, Asterios Katsifodimos, Tilmann Rabl, Volker Markl

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, or minimizing memory usage. However, each technique operates under different assumptions with respect to workload characteristics, such as properties of aggregation functions (e.g., invertible, associative), window types (e.g., sliding, sessions), windowing measures (e.g., time-or count-based), and stream (dis)order. In this article, we present Scotty, an efficient and general open-source operator for sliding-window aggregation in stream processing systems, such as Apache Flink, Apache Beam, Apache Samza, Apache Kafka, Apache Spark, and Apache Storm. One can easily extend Scotty with user-defined aggregation functions and window types. Scotty implements the concept of general stream slicing and derives workload characteristics from aggregation queries to improve performance without sacrificing its general applicability. We provide an in-depth view on the algorithms of the general stream slicing approach. Our experiments show that Scotty outperforms alternative solutions.

Original languageEnglish
Article number1
Number of pages46
JournalACM Transactions on Database Systems
Volume46
Issue number1
DOIs
Publication statusPublished - 2021

Keywords

  • aggregate sharing
  • aggregation
  • Apache Beam
  • Apache Flink
  • Apache Kafka Streams
  • Apache Samza
  • Apache Spark
  • Apache Storm
  • open-source
  • Scotty
  • session window
  • sliding-window
  • stream processing
  • tumbling window
  • Window

Fingerprint

Dive into the research topics of 'Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems'. Together they form a unique fingerprint.

Cite this