Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks

Michael Riegler, Vamsidhar Reddy Gaddam, Martha Larson, Ragnhild Eg, Pål Halvorsen, Carsten Griwodz

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

7 Citations (Scopus)

Abstract

Crowdsourcing has established itself as a powerful tool for multimedia researchers, and is commonly used to collect human input for various purposes. It is also a fairly widespread practice to control the contributions of users based on the quality of their input. This paper points to the fact that applying this practice in subjective assessment tasks may lead to an undesired, negative outcome. We present a crowdsourcing experiment and a discussion of the ways in which control in crowdsourcing studies can lead to a phenomenon akin to a self-fulfilling prophecy. This paper is intended to trigger discussion and lead to more deeply reflective crowdsourcing practices in the multimedia context.
Original languageEnglish
Title of host publication2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI)
PublisherIEEE
Pages1-6
Number of pages6
ISBN (Electronic)978-1-4673-8695-1
DOIs
Publication statusPublished - 2016
Event2016 14th International Workshop on Content-Based Multimedia Indexing - Bucharest, Romania
Duration: 15 Jun 201617 Jun 2016
http://cbmi2016.upb.ro/

Conference

Conference2016 14th International Workshop on Content-Based Multimedia Indexing
Abbreviated titleCBMI
Country/TerritoryRomania
CityBucharest
Period15/06/1617/06/16
Internet address

Keywords

  • Crowdsourcing
  • Multimedia communication
  • Quality of service
  • Cameras
  • Distortion
  • Standards
  • Physiology

Fingerprint

Dive into the research topics of 'Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks'. Together they form a unique fingerprint.

Cite this