Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers

Long Cheng, Ying Wang, Qingzhi Liu, Dick H.J. Epema, Cheng Liu, Ying Mao, John Murphy

Research output: Contribution to journalArticleScientificpeer-review

1 Citation (Scopus)
5 Downloads (Pure)

Abstract

Large data centers are currently the mainstream infrastructures for big data processing. As one of the most fundamental tasks in these environments, the efficient execution of distributed data operators (e.g., join and aggregation) are still challenging current data systems, and one of the key performance issues is network communication time. State-of-the-art methods trying to improve that problem focus on either application-layer data locality optimization to reduce network traffic or on network-layer data flow optimization to increase bandwidth utilization. However, the techniques in the two layers are totally independent from each other, and performance gains from a joint optimization perspective have not yet been explored. In this article, we propose a novel approach called NEAL (NEtwork-Aware Locality scheduling) to bridge this gap, and consequently to further reduce communication time for distributed big data operators. We present the detailed design and implementation of NEAL, and our experimental results demonstrate that NEAL always performs better than current approaches for different workloads and network bandwidth configurations.

Original languageEnglish
Article number9329172
Pages (from-to)1494-1510
Number of pages17
JournalIEEE Transactions on Parallel and Distributed Systems
Volume32
Issue number6
DOIs
Publication statusPublished - 2021

Bibliographical note

Accepted author manuscript

Keywords

  • Data locality
  • SDN
  • big data
  • coflow scheduling
  • data centers
  • distributed operators
  • metaheuristic

Fingerprint

Dive into the research topics of 'Network-Aware Locality Scheduling for Distributed Data Operators in Data Centers'. Together they form a unique fingerprint.

Cite this