Crowd-Powered Hybrid Classification Services: Calibration is all you need

Burcu Sayin, Evgeny Krivosheev, Jorge Ramirez, Fabio Casati, Ekaterina Taran, Veronika Malanina, Jie Yang

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

Hybrid classification services are online services that combine machine learning (ML) and humans - either crowd workers or experts - to achieve a classification objective, from relatively simple ones such as deriving the sentiment of a text to more complex ones such as medical diagnoses. This paper takes the first steps toward a science for hybrid classification services, discussing key concepts, challenges, and architectures, and then focusing on a central aspect, that of ML calibration and how it can be achieved with crowdsourced labels.
Original languageEnglish
Title of host publication2021 IEEE International Conference on Web Services (ICWS)
EditorsL. O'Conner
Place of PublicationPiscataway
PublisherIEEE
Pages42-50
Number of pages9
ISBN (Electronic)978-1-6654-1681-8
ISBN (Print)978-1-6654-1682-5
DOIs
Publication statusPublished - 2021
EventICWS 2021: 2021 IEEE International Conference on Web Services - Virtual at Chicago, United States
Duration: 5 Sep 202110 Sep 2021

Conference

ConferenceICWS 2021
CountryUnited States
CityVirtual at Chicago
Period5/09/2110/09/21

Keywords

  • crowdsourcing
  • machine learning
  • Calibration

Fingerprint

Dive into the research topics of 'Crowd-Powered Hybrid Classification Services: Calibration is all you need'. Together they form a unique fingerprint.

Cite this