EFE: End-to-end Frame-to-Gaze Estimation

Haldun  Balim; Seonwook  Park; Xi Wang; Xucong Zhang; Otmar Hilliges

doi:10.1109/CVPRW59228.2023.00269

EFE: End-to-end Frame-to-Gaze Estimation

Haldun Balim, Seonwook Park, Xi Wang, Xucong Zhang, Otmar Hilliges

Pattern Recognition and Bioinformatics

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

4 Citations (Scopus)

33 Downloads (Pure)

Abstract

Despite the recent development of learning-based gaze estimation methods, most methods require one or more eye or face region crops as inputs and produce a gaze direction vector as output. Cropping results in a higher resolution in the eye regions and having fewer confounding factors (such as clothing and hair) is believed to benefit the final model performance. However, this eye/face patch cropping process is expensive, erroneous, and implementation-specific for different methods. In this paper, we propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping. Our method demonstrates that direct gaze regression from the raw downscaled frame, from FHD/HD to VGA/HVGA resolution, is possible despite the challenges of having very few pixels in the eye region. The proposed method achieves comparable results to state-of-the-art methods in Point-of-Gaze (PoG) estimation on three public gaze datasets: GazeCapture, MPIIFaceGaze, and EVE, and generalizes well to extreme camera view changes.

Original language	English
Title of host publication	Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Editors	Lisa O’Conner
Place of Publication	Piscataway
Publisher	IEEE
Pages	2688-2697
Number of pages	10
ISBN (Electronic)	979-8-3503-0249-3
ISBN (Print)	979-8-3503-0250-9
DOIs	https://doi.org/10.1109/CVPRW59228.2023.00269
Publication status	Published - 2023
Event	2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) - Vancouver, Canada Duration: 17 Jun 2023 → 24 Jun 2023

Publication series

Name	IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops
Volume	2023-June
ISSN (Print)	2160-7508
ISSN (Electronic)	2160-7516

Conference

Conference	2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
Country/Territory	Canada
City	Vancouver
Period	17/06/23 → 24/06/23

Bibliographical note

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Access to Document

10.1109/CVPRW59228.2023.00269

EFE_End-to-end_Frame-to-Gaze_EstimationFinal published version, 3.93 MB

Cite this

Balim, H., Park, S., Wang, X., Zhang, X., & Hilliges, O. (2023). EFE: End-to-end Frame-to-Gaze Estimation. In L. O’Conner (Ed.), Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 2688-2697). (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops; Vol. 2023-June). IEEE. https://doi.org/10.1109/CVPRW59228.2023.00269

@inproceedings{3ee9d60136ea4e318878524a58a6b0de,

title = "EFE: End-to-end Frame-to-Gaze Estimation",

abstract = "Despite the recent development of learning-based gaze estimation methods, most methods require one or more eye or face region crops as inputs and produce a gaze direction vector as output. Cropping results in a higher resolution in the eye regions and having fewer confounding factors (such as clothing and hair) is believed to benefit the final model performance. However, this eye/face patch cropping process is expensive, erroneous, and implementation-specific for different methods. In this paper, we propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping. Our method demonstrates that direct gaze regression from the raw downscaled frame, from FHD/HD to VGA/HVGA resolution, is possible despite the challenges of having very few pixels in the eye region. The proposed method achieves comparable results to state-of-the-art methods in Point-of-Gaze (PoG) estimation on three public gaze datasets: GazeCapture, MPIIFaceGaze, and EVE, and generalizes well to extreme camera view changes.",

author = "Haldun Balim and Seonwook Park and Xi Wang and Xucong Zhang and Otmar Hilliges",

note = "Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.; 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) ; Conference date: 17-06-2023 Through 24-06-2023",

year = "2023",

doi = "10.1109/CVPRW59228.2023.00269",

language = "English",

isbn = "979-8-3503-0250-9",

series = "IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops",

publisher = "IEEE",

pages = "2688--2697",

editor = "{O{\textquoteright}Conner }, {Lisa }",

booktitle = "Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)",

address = "United States",

}

Balim, H, Park, S, Wang, X, Zhang, X & Hilliges, O 2023, EFE: End-to-end Frame-to-Gaze Estimation. in L O’Conner (ed.), Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, vol. 2023-June, IEEE, Piscataway, pp. 2688-2697, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, Canada, 17/06/23. https://doi.org/10.1109/CVPRW59228.2023.00269

EFE: End-to-end Frame-to-Gaze Estimation. / Balim, Haldun ; Park, Seonwook ; Wang, Xi et al.
Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). ed. / Lisa O’Conner . Piscataway: IEEE, 2023. p. 2688-2697 (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops; Vol. 2023-June).

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - EFE: End-to-end Frame-to-Gaze Estimation

AU - Balim, Haldun

AU - Park, Seonwook

AU - Wang, Xi

AU - Zhang, Xucong

AU - Hilliges, Otmar

N1 - Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2023

Y1 - 2023

N2 - Despite the recent development of learning-based gaze estimation methods, most methods require one or more eye or face region crops as inputs and produce a gaze direction vector as output. Cropping results in a higher resolution in the eye regions and having fewer confounding factors (such as clothing and hair) is believed to benefit the final model performance. However, this eye/face patch cropping process is expensive, erroneous, and implementation-specific for different methods. In this paper, we propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping. Our method demonstrates that direct gaze regression from the raw downscaled frame, from FHD/HD to VGA/HVGA resolution, is possible despite the challenges of having very few pixels in the eye region. The proposed method achieves comparable results to state-of-the-art methods in Point-of-Gaze (PoG) estimation on three public gaze datasets: GazeCapture, MPIIFaceGaze, and EVE, and generalizes well to extreme camera view changes.

AB - Despite the recent development of learning-based gaze estimation methods, most methods require one or more eye or face region crops as inputs and produce a gaze direction vector as output. Cropping results in a higher resolution in the eye regions and having fewer confounding factors (such as clothing and hair) is believed to benefit the final model performance. However, this eye/face patch cropping process is expensive, erroneous, and implementation-specific for different methods. In this paper, we propose a frame-to-gaze network that directly predicts both 3D gaze origin and 3D gaze direction from the raw frame out of the camera without any face or eye cropping. Our method demonstrates that direct gaze regression from the raw downscaled frame, from FHD/HD to VGA/HVGA resolution, is possible despite the challenges of having very few pixels in the eye region. The proposed method achieves comparable results to state-of-the-art methods in Point-of-Gaze (PoG) estimation on three public gaze datasets: GazeCapture, MPIIFaceGaze, and EVE, and generalizes well to extreme camera view changes.

UR - http://www.scopus.com/inward/record.url?scp=85170827245&partnerID=8YFLogxK

U2 - 10.1109/CVPRW59228.2023.00269

DO - 10.1109/CVPRW59228.2023.00269

M3 - Conference contribution

SN - 979-8-3503-0250-9

T3 - IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops

SP - 2688

EP - 2697

BT - Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

A2 - O’Conner , Lisa

PB - IEEE

CY - Piscataway

T2 - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Y2 - 17 June 2023 through 24 June 2023

ER -

Balim H, Park S, Wang X, Zhang X, Hilliges O. EFE: End-to-end Frame-to-Gaze Estimation. In O’Conner L, editor, Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway: IEEE. 2023. p. 2688-2697. (IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops). doi: 10.1109/CVPRW59228.2023.00269

EFE: End-to-end Frame-to-Gaze Estimation

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this