Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks

Michael Riegler; Vamsidhar Reddy Gaddam; Martha Larson; Ragnhild Eg; Pål Halvorsen; Carsten Griwodz

doi:10.1109/cbmi.2016.7500256

Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks

Michael Riegler, Vamsidhar Reddy Gaddam, Martha Larson, Ragnhild Eg, Pål Halvorsen, Carsten Griwodz

Multimedia Computing

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

8 Citations (Scopus)

Abstract

Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.

Original language	English
Title of host publication	2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI)
Publisher	IEEE
Pages	1-6
Number of pages	6
ISBN (Electronic)	978-1-4673-8695-1
DOIs	https://doi.org/10.1109/cbmi.2016.7500256
Publication status	Published - 2016
Event	2016 14th International Workshop on Content-Based Multimedia Indexing - Bucharest, Romania Duration: 15 Jun 2016 → 17 Jun 2016 http://cbmi2016.upb.ro/

Conference

Conference	2016 14th International Workshop on Content-Based Multimedia Indexing
Abbreviated title	CBMI
Country/Territory	Romania
City	Bucharest
Period	15/06/16 → 17/06/16
Internet address	http://cbmi2016.upb.ro/

Keywords

Crowdsourcing
Multimedia communication
Quality of service
Cameras
Distortion
Standards
Physiology

Access to Document

10.1109/cbmi.2016.7500256

Cite this

@inproceedings{2db223c77d5940d28fdd30e02a56c937,

title = "Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks",

abstract = "Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.",

keywords = "Crowdsourcing, Multimedia communication, Quality of service, Cameras, Distortion, Standards, Physiology ",

author = "Michael Riegler and Gaddam, {Vamsidhar Reddy} and Martha Larson and Ragnhild Eg and P{\aa}l Halvorsen and Carsten Griwodz",

year = "2016",

doi = "10.1109/cbmi.2016.7500256",

language = "English",

pages = "1--6",

booktitle = "2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI)",

publisher = "IEEE",

address = "United States",

note = "2016 14th International Workshop on Content-Based Multimedia Indexing, CBMI ; Conference date: 15-06-2016 Through 17-06-2016",

url = "http://cbmi2016.upb.ro/",

}

Riegler, M, Gaddam, VR, Larson, M, Eg, R, Halvorsen, P & Griwodz, C 2016, Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks. in 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI). IEEE, pp. 1-6, 2016 14th International Workshop on Content-Based Multimedia Indexing, Bucharest, Romania, 15/06/16. https://doi.org/10.1109/cbmi.2016.7500256

Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks. / Riegler, Michael; Gaddam, Vamsidhar Reddy; Larson, Martha et al.
2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI). IEEE, 2016. p. 1-6.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - Crowdsourcing as self-fulfilling prophecy

T2 - 2016 14th International Workshop on Content-Based Multimedia Indexing

AU - Riegler, Michael

AU - Gaddam, Vamsidhar Reddy

AU - Larson, Martha

AU - Eg, Ragnhild

AU - Halvorsen, Pål

AU - Griwodz, Carsten

PY - 2016

Y1 - 2016

N2 - Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.

AB - Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.

KW - Crowdsourcing

KW - Multimedia communication

KW - Quality of service

KW - Cameras

KW - Distortion

KW - Standards

KW - Physiology

U2 - 10.1109/cbmi.2016.7500256

DO - 10.1109/cbmi.2016.7500256

M3 - Conference contribution

SP - 1

EP - 6

BT - 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI)

PB - IEEE

Y2 - 15 June 2016 through 17 June 2016

ER -

Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this