Crowd-Powered Hybrid Classification Services: Calibration is all you need

Burcu Sayin, Evgeny Krivosheev, Jorge Ramirez, Fabio Casati, Ekaterina Taran, Veronika Malanina, Jie Yang

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review


Hybrid classification services are online services that combine machine learning (ML) and humans - either crowd workers or experts - to achieve a classification objective, from relatively simple ones such as deriving the sentiment of a text to more complex ones such as medical diagnoses. This paper takes the first steps toward a science for hybrid classification services, discussing key concepts, challenges, and architectures, and then focusing on a central aspect, that of ML calibration and how it can be achieved with crowdsourced labels.
Original languageEnglish
Title of host publication2021 IEEE International Conference on Web Services (ICWS)
EditorsL. O'Conner
Place of PublicationPiscataway
Number of pages9
ISBN (Electronic)978-1-6654-1681-8
ISBN (Print)978-1-6654-1682-5
Publication statusPublished - 2021
EventICWS 2021: 2021 IEEE International Conference on Web Services - Virtual at Chicago, United States
Duration: 5 Sep 202110 Sep 2021


ConferenceICWS 2021
CountryUnited States
CityVirtual at Chicago


  • crowdsourcing
  • machine learning
  • Calibration


Dive into the research topics of 'Crowd-Powered Hybrid Classification Services: Calibration is all you need'. Together they form a unique fingerprint.

Cite this