Hazelcast jet: Low-latency stream processing at the 99.99th percentile

Can Gencer, Marko Topolnik, Viliam Ďurina, Emin Demirci, Ensar B. Kahveci, Ali Gürbüz, Ondřej Lukáš, Marios Fragkoulis, Asterios Katsifodimos, More Authors

Research output: Contribution to journalConference articleScientificpeer-review

1 Citation (Scopus)
15 Downloads (Pure)

Abstract

Jet is an open-source, high-performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet’s purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast’s In-memory Data Grid (IMDG): a set of in-memory, partitioned and replicated data structures. With time, Jet evolved into a full-fledged, scale-out stream processor that can handle out-of-order streams and provide exactly-once processing guarantees. Jet’s end-to-end latency lies in the order of milliseconds, and its throughput in the order of millions of events per CPU-core. This paper presents the main design decisions we made in order to maximize the performance per CPU-core, alongside lessons learned, and an empirical performance evaluation.

Original languageEnglish
Pages (from-to)3110-3121
Number of pages12
JournalProceedings of the VLDB Endowment
Volume14
Issue number12
DOIs
Publication statusPublished - 2021
Event47th International Conference on Very Large Data Bases, VLDB 2021 - Virtual, Online
Duration: 16 Aug 202120 Aug 2021

Fingerprint

Dive into the research topics of 'Hazelcast jet: Low-latency stream processing at the 99.99th percentile'. Together they form a unique fingerprint.

Cite this