Masa: Responsive Multi-DNN Inference on the Edge

Bart Cox, Jeroen Galjaard, Amirmasoud Ghiassi, Robert Birke, Lydia Y. Chen

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

12 Citations (Scopus)

Abstract

Deep neural networks (DNNs) are becoming the core components of many applications running on edge devices, especially for real time image-based analysis. Increasingly, multi-faced knowledge is extracted via executing multiple DNNs inference models, e.g., identifying objects, faces, and genders from images. The response times of multi-DNN highly affect users' quality of experience and safety as well. Different DNNs exhibit diversified resource requirements and execution patterns across layers and networks, which may easily exceed the available device memory and riskily degrade the responsiveness. In this paper, we design and implement Masa, a responsive memory-aware multi-DNN execution framework, an on-device middleware featuring on modeling inter- and intra-network dependency and leveraging complimentary memory usage of each layer. Masa can consistently ensure the average response time when deterministically and stochastically executing multiple DNN-based image analyses. We extensively evaluate Masa on three configurations of Raspberry Pi and a large set of popular DNN models triggered by different generation patterns of images. Our evaluation results show that Masa can achieve lower average response times by up to 90% on devices with small memory, i.e., 512 MB to 1 GB, compared to the state of the art multi-DNN scheduling solutions.

Original languageEnglish
Title of host publication2021 IEEE International Conference on Pervasive Computing and Communications, PerCom 2021
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)9781665404181
DOIs
Publication statusPublished - 2021
Event19th IEEE International Conference on Pervasive Computing and Communications, PerCom 2021 - Virtual, Kassel, Germany
Duration: 22 Mar 202126 Mar 2021

Publication series

Name2021 IEEE International Conference on Pervasive Computing and Communications, PerCom 2021

Conference

Conference19th IEEE International Conference on Pervasive Computing and Communications, PerCom 2021
Country/TerritoryGermany
CityVirtual, Kassel
Period22/03/2126/03/21

Keywords

  • average response time
  • edge devices
  • memory-aware scheduling
  • Multiple DNNs inference

Fingerprint

Dive into the research topics of 'Masa: Responsive Multi-DNN Inference on the Edge'. Together they form a unique fingerprint.

Cite this