Achieving Performance Balance among Spark Frameworks with Two-Level Schedulers

Aleksandra Kuzmanovska, Hans van den Bogert, Rudolf Mak, Dick Epema

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

2 Citations (Scopus)


When multiple data-processing frameworks with time-varying workloads are simultaneously present in a single cluster or data-center, an apparent goal is to have them experience equal performance, expressed in whatever performance metrics are applicable. In modern data-center environments, Two-Level Schedulers (TLSs) that leave the scheduling of individual jobs to the schedulers within the data-processing frameworks are typically used for managing the resources of data-processing frameworks. Two such TLSs with opposite designs are Mesos and Koala-F. Mesos employs fine-grained resource allocation and aims at Dominant Resource Fairness (DRF) among framework instances by offering resources to them for the duration of a single task. In contrast, Koala-F aims at performance fairness among framework instances by employing dynamic coarse-grained resource allocation of sets of complete nodes based on performance feedback from individual instances. The goal of this paper is to explore the trade-offs between these two TLS designs when trying to achieve performance balance among frameworks. We select Apache Spark as a representative of data-processing frameworks, and perform experiments on a modest-sized cluster, using jobs chosen from commonly used data-processing benchmarks. Our results reveal that achieving performance balance among framework instances is a challenge for both TLS designs, despite their opposite design choices. Moreover, we exhibit design flaws in the DRF allocation policy that prevent Mesos from achieving performance balance. Finally, to remedy these flaws, we propose a feedback controller for Mesos that dynamically adapts framework weights, as used in Weighted DRF (W-DRF), based on their performance.

Original languageEnglish
Title of host publicationProceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018
Place of PublicationLos Alamitos, CA
Number of pages10
ISBN (Electronic)9781538658154
Publication statusPublished - 2018
EventCCGRID 2018: 18th IEEE/ACM International Sympocium on Cluster, Cloud and Grid Computing - Washington, DC, United States
Duration: 1 May 20184 May 2018
Conference number: 18


ConferenceCCGRID 2018
Country/TerritoryUnited States
CityWashington, DC


  • Data processing framework
  • DRF
  • Job slowdown
  • Koala F
  • Mesos
  • Performance balance
  • Resource allocation policy
  • Spark
  • Two level schedulers


Dive into the research topics of 'Achieving Performance Balance among Spark Frameworks with Two-Level Schedulers'. Together they form a unique fingerprint.

Cite this