Generative RGB-D face completion for head-mounted display removal

Nels Numan*, Frank Ter Haar, Pablo Cesar

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

11 Citations (Scopus)

Abstract

Head-mounted displays (HMDs) are an essential display device for the observation of virtual reality (VR) environments. However, HMDs obstruct external capturing methods from recording the user's upper face. This severely impacts social VR applications, such as teleconferencing, which commonly rely on external RGB-D sensors to capture a volumetric representation of the user. In this paper, we introduce an HMD removal framework based on generative adversarial networks (GANs), capable of jointly filling in missing color and depth data in RGB-D face images. Our framework includes an RGB-based identity loss function for identity preservation and several components aimed at surface reproduction. Our results demonstrate that our framework is able to remove HMDs from synthetic RGB-D face images while preserving the subject's identity.

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021
PublisherIEEE
Pages109-116
Number of pages8
ISBN (Electronic)978-3-7381-1367-8
DOIs
Publication statusPublished - 2021
Event2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021 - Virtual, Lisbon, Portugal
Duration: 27 Mar 20213 Apr 2021

Publication series

NameProceedings - 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021

Conference

Conference2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2021
Country/TerritoryPortugal
CityVirtual, Lisbon
Period27/03/213/04/21

Keywords

  • Artificial intelligence
  • Computer vision
  • Computing methodologies
  • Human computer interaction (HCI)
  • Interaction paradigms
  • Reconstruction; Human-centered computing
  • Virtual reality

Fingerprint

Dive into the research topics of 'Generative RGB-D face completion for head-mounted display removal'. Together they form a unique fingerprint.

Cite this