A Coflow-based Co-optimization Framework for High-performance Data Analytics

Long Cheng, Ying Wang, Yulong Pei, Dick Epema

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

9 Citations (Scopus)
44 Downloads (Pure)

Abstract

Efficient execution of distributed database operators such as joining and aggregating is critical for the performance of big data analytics. With the increase of the compute speedup of modern CPUs, reducing the network
communication time of these operators in large systems is becoming increasingly important, and also challenging current techniques. Significant performance improvements have been achieved by using state-of-the-art methods, such as reducing network traffic designed in the data management domain, and data flow scheduling in the data communications domain.
However, the proposed techniques in both fields just view each other as a black box, and performance gains from a co-optimization perspective have not yet been explored.
In this paper, based on current research in coflow scheduling,
we propose a novel Coflow-based Co-optimization Framework
(CCF), which can co-optimize application-level data movement
and network-level data communications for distributed operators,
and consequently contribute to their performance in
large distributed environments. We present the detailed design
and implementation of CCF, and conduct an experimental
evaluation of CCF using large-scale simulations on large data
joins. Our results demonstrate that CCF can always perform
faster than current approaches on network communications in
large-scale distributed scenarios.
Original languageEnglish
Title of host publicationProceedings - 46th International Conference on Parallel Processing, ICPP 2017
Place of PublicationLos Alamitos, CA
PublisherIEEE
Pages392-401
Number of pages10
ISBN (Electronic)978-1-5386-1042-8
DOIs
Publication statusPublished - 2017
EventICPP 2017: 46th International Conference on Parallel Processing - Bristol, United Kingdom
Duration: 14 Aug 201717 Aug 2017
Conference number: 46
http://www.icpp-conf.org/2017/index.php

Conference

ConferenceICPP 2017
CountryUnited Kingdom
CityBristol
Period14/08/1717/08/17
Internet address

Keywords

  • big data
  • coflow scheduling
  • distributed joins
  • network communications
  • data-intensive applications

Fingerprint Dive into the research topics of 'A Coflow-based Co-optimization Framework for High-performance Data Analytics'. Together they form a unique fingerprint.

  • Cite this

    Cheng, L., Wang, Y., Pei, Y., & Epema, D. (2017). A Coflow-based Co-optimization Framework for High-performance Data Analytics. In Proceedings - 46th International Conference on Parallel Processing, ICPP 2017 (pp. 392-401). IEEE. https://doi.org/10.1109/ICPP.2017.48