Temporal interpolation of dynamic digital humans using convolutional neural networks

Irene Viola, Jelmer Mulder, Francesca De Simone, Pablo Cesar

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

In recent years, there has been an increased interest in point cloud representation for visualizing digital humans in cross reality. However, due to their voluminous size, point clouds require high bandwidth to be transmitted. In this paper, we propose a temporal interpolation architecture capable of increasing the temporal resolution of dynamic digital humans, represented using point clouds. With this technique, bandwidth savings can be achieved by transmitting dynamic point clouds in a lower temporal resolution, and recreating a higher temporal resolution on the receiving side. Our interpolation architecture works by first downsampling the point clouds to a lower spatial resolution, then estimating scene flow using a newly designed neural network architecture, and finally upsampling the result back to the original spatial resolution. To improve the smoothness of the results, we additionally apply a novel technique called neighbour snapping. To be able to train and test our newly designed network, we created a synthetic point cloud data set of animated human bodies. Results from the evaluation of our architecture through a small-scale user study show the benefits of our method with respect to the state of the art in scene flow estimation for point clouds. Moreover, correlation between our user study and existing objective quality metrics confirm the need for new metrics to accurately predict the visual quality of point cloud contents.

Original languageEnglish
Title of host publicationProceedings - 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Pages90-97
Number of pages8
ISBN (Electronic)9781728156040
DOIs
Publication statusPublished - 2019
Event2nd IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019 - San Diego, United States
Duration: 9 Dec 201911 Dec 2019

Conference

Conference2nd IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019
CountryUnited States
CitySan Diego
Period9/12/1911/12/19

Keywords

  • Cross reality
  • Digital humans
  • Point cloud
  • Scene flow estimation
  • Temporal interpolation

Fingerprint Dive into the research topics of 'Temporal interpolation of dynamic digital humans using convolutional neural networks'. Together they form a unique fingerprint.

  • Cite this

    Viola, I., Mulder, J., De Simone, F., & Cesar, P. (2019). Temporal interpolation of dynamic digital humans using convolutional neural networks. In Proceedings - 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality, AIVR 2019 (pp. 90-97). [8942286] Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/AIVR46125.2019.00022