Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

39 Citations (Scopus)

Abstract

Event cameras are novel vision sensors that sample, in an asynchronous fashion, brightness increments with low latency and high temporal resolution. The resulting streams of events are of high value by themselves, especially for high speed motion estimation. However, a growing body of work has also focused on the reconstruction of intensity frames from the events, as this allows bridging the gap with the existing literature on appearance- and frame-based computer vision. Recent work has mostly approached this problem using neural networks trained with synthetic, ground-truth data. In this work we approach, for the first time, the intensity reconstruction problem from a self-supervised learning perspective. Our method, which leverages the knowledge of the inner workings of event cameras, combines estimated optical flow and the event-based photometric constancy to train neural networks without the need for any ground-truth or synthetic data. Results across multiple datasets show that the performance of the proposed self-supervised approach is in line with the state-of-the-art. Additionally, we propose a novel, lightweight neural network for optical flow estimation that achieves high speed inference with only a minor drop in performance.
Original languageEnglish
Title of host publication2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Subtitle of host publicationProceedings
Place of PublicationPiscataway
PublisherIEEE
Pages3445-3454
Number of pages10
ISBN (Electronic)978-1-6654-4509-2
ISBN (Print)978-1-6654-4510-8
DOIs
Publication statusPublished - 2021
Event2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition
- Virtual at Nashville, United States
Duration: 20 Jun 202125 Jun 2021

Publication series

NameProceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
ISSN (Print)1063-6919

Conference

Conference2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition
Abbreviated titleCVPR 2021
Country/TerritoryUnited States
CityVirtual at Nashville
Period20/06/2125/06/21

Fingerprint

Dive into the research topics of 'Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy'. Together they form a unique fingerprint.

Cite this