Accelerating Gossip-Based Deep Learning in Heterogeneous Edge Computing Platforms

Rui Han, Shilin Li, Xiangwei Wang, Chi Harold Liu, Gaofeng Xin, Lydia Y. Chen

Research output: Contribution to journalArticleScientificpeer-review

Abstract

With the exponential growth of data created at the network edge, decentralized and Gossip-based training of deep learning (DL) models on edge computing (EC) gains tremendous research momentum, owing to its capability to learn from resource-strenuous edge nodes with limited network connectivity. Today's edge devices are extremely heterogeneous, e.g., hardware and software stacks, and result in high performance variation of training time and inducing extra delay to synchronize and converge. The large body of prior art accelerates DL, being data or model parallelization, via a centralized server, e.g., parameter server scheme, which may easily turn into the system bottleneck or single point of failure. In this artice, we propose EdgeGossip, a framework specifically designed to accelerate the training process of decentralized and Gossip-based DL training for heterogeneous EC platforms. EdgeGossip features on: (i) low performance variation among multiple EC platforms during iterative training, and (ii) accuracy-aware training to fastly obtain best possible model accuracy. We implement EdgeGossip based on popular Gossip algorithms and demonstrate its effectiveness using real-world DL workloads, i.e., considerably reducing model training time by an average of 2.70 times while only incurring accuracy losses of 0.78 percent.

Original languageEnglish
Article number9303468
Pages (from-to)1591-1602
Number of pages12
JournalIEEE Transactions on Parallel and Distributed Systems
Volume32
Issue number7
DOIs
Publication statusPublished - 2021

Keywords

  • decentralized training
  • Deep learning
  • edge computing
  • gossip

Fingerprint Dive into the research topics of 'Accelerating Gossip-Based Deep Learning in Heterogeneous Edge Computing Platforms'. Together they form a unique fingerprint.

Cite this