Humans Disagree With the IoU for Measuring Object Detector Localization Error

Ombretta Strafforello; Vanathi Rajasekart; Osman S. Kayhan; Oana  Inel; Jan van Gemert

doi:10.1109/ICIP46576.2022.9898043

Humans Disagree With the IoU for Measuring Object Detector Localization Error

Ombretta Strafforello, Vanathi Rajasekart, Osman S. Kayhan, Oana Inel, Jan van Gemert

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

42 Downloads (Pure)

Abstract

The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.

Original language	English
Title of host publication	Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP)
Place of Publication	Piscataway
Publisher	IEEE
Pages	1261-1265
Number of pages	5
ISBN (Electronic)	978-1-6654-9620-9
ISBN (Print)	978-1-6654-9621-6
DOIs	https://doi.org/10.1109/ICIP46576.2022.9898043
Publication status	Published - 2022
Event	2022 IEEE International Conference on Image Processing (ICIP) - Bordeaux, France Duration: 16 Oct 2022 → 19 Oct 2022

Conference

Conference	2022 IEEE International Conference on Image Processing (ICIP)
Country/Territory	France
City	Bordeaux
Period	16/10/22 → 19/10/22

Bibliographical note

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

object detection
IoU
human preference

Access to Document

10.1109/ICIP46576.2022.9898043

Humans_Disagree_With_the_IoU_for_Measuring_Object_Detector_Localization_ErrorFinal published version, 5.57 MB

Cite this

@inproceedings{7bb5011f30244fbaad4f84b1064568cd,

title = "Humans Disagree With the IoU for Measuring Object Detector Localization Error",

abstract = "The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.",

keywords = "object detection, IoU, human preference",

author = "Ombretta Strafforello and Vanathi Rajasekart and Kayhan, {Osman S.} and Oana Inel and {van Gemert}, Jan",

note = "Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.; 2022 IEEE International Conference on Image Processing (ICIP) ; Conference date: 16-10-2022 Through 19-10-2022",

year = "2022",

doi = "10.1109/ICIP46576.2022.9898043",

language = "English",

isbn = "978-1-6654-9621-6",

pages = "1261--1265",

booktitle = "Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP)",

publisher = "IEEE",

address = "United States",

}

Strafforello, O, Rajasekart, V, Kayhan, OS, Inel, O & van Gemert, J 2022, Humans Disagree With the IoU for Measuring Object Detector Localization Error. in Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP). IEEE, Piscataway, pp. 1261-1265, 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 16/10/22. https://doi.org/10.1109/ICIP46576.2022.9898043

Humans Disagree With the IoU for Measuring Object Detector Localization Error. / Strafforello, Ombretta; Rajasekart, Vanathi; Kayhan, Osman S. et al.
Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP). Piscataway: IEEE, 2022. p. 1261-1265.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Humans Disagree With the IoU for Measuring Object Detector Localization Error

AU - Strafforello, Ombretta

AU - Rajasekart, Vanathi

AU - Kayhan, Osman S.

AU - Inel, Oana

AU - van Gemert, Jan

N1 - Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2022

Y1 - 2022

N2 - The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.

AB - The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.

KW - object detection

KW - IoU

KW - human preference

UR - http://www.scopus.com/inward/record.url?scp=85146669015&partnerID=8YFLogxK

U2 - 10.1109/ICIP46576.2022.9898043

DO - 10.1109/ICIP46576.2022.9898043

M3 - Conference contribution

SN - 978-1-6654-9621-6

SP - 1261

EP - 1265

BT - Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP)

PB - IEEE

CY - Piscataway

T2 - 2022 IEEE International Conference on Image Processing (ICIP)

Y2 - 16 October 2022 through 19 October 2022

ER -

Humans Disagree With the IoU for Measuring Object Detector Localization Error

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Code underlying the publication: "Humans disagree with the IoU for measuring object detector localization error"

Cite this

Humans Disagree With the IoU for Measuring Object Detector Localization Error

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Datasets

Code underlying the publication: "Humans disagree with the IoU for measuring object detector localization error"

Cite this