Crowd-Powered Hybrid Classification Services: Calibration is all you need

Burcu Sayin, Evgeny Krivosheev, Jorge Ramirez, Fabio Casati, Ekaterina Taran, Veronika Malanina, Jie Yang

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

2 Citations (Scopus)

Abstract

Hybrid classification services are online services that combine machine learning (ML) and humans - either crowd workers or experts - to achieve a classification objective, from relatively simple ones such as deriving the sentiment of a text to more complex ones such as medical diagnoses. This paper takes the first steps toward a science for hybrid classification services, discussing key concepts, challenges, and architectures, and then focusing on a central aspect, that of ML calibration and how it can be achieved with crowdsourced labels.
Original languageEnglish
Title of host publication2021 IEEE International Conference on Web Services (ICWS)
EditorsCarl K. Chang, Ernesto Damiani, Jing Fan, Parisa Ghodous, Michael Maximilien, Zhongjie Wang, Robert Ward, Jia Zhang
Place of PublicationPiscataway
PublisherIEEE
Pages42-50
Number of pages9
ISBN (Electronic)978-1-6654-1681-8
ISBN (Print)978-1-6654-1682-5
DOIs
Publication statusPublished - 2021
EventICWS 2021: 2021 IEEE International Conference on Web Services - Virtual at Chicago, United States
Duration: 5 Sept 202110 Sept 2021

Publication series

NameProceedings - 2021 IEEE International Conference on Web Services, ICWS 2021

Conference

ConferenceICWS 2021
Country/TerritoryUnited States
CityVirtual at Chicago
Period5/09/2110/09/21

Keywords

  • Calibration
  • crowdsourcing
  • machine learning

Fingerprint

Dive into the research topics of 'Crowd-Powered Hybrid Classification Services: Calibration is all you need'. Together they form a unique fingerprint.

Cite this