HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale

Sepideh Mesbah*, Ines Arous, Jie Yang, Alessandro Bozzon*

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

56 Downloads (Pure)


Evaluating design ideas is necessary to predict their success and assess their impact early on in the process. Existing methods rely either on metrics computed by systems that are effective but subject to errors and bias, or experts' ratings, which are accurate but expensive and long to collect. Crowdsourcing offers a compelling way to evaluate a large number of design ideas in a short amount of time while being cost-effective. Workers' evaluation is, however, less reliable and might substantially differ from experts' evaluation. In this work, we investigate workers' rating behavior and compare it with experts. First, we instrument a crowdsourcing study where we asked workers to evaluate design ideas from three innovation challenges. We show that workers share similar insights with experts but tend to rate more generously and weigh certain criteria more importantly. Next, we develop a hybrid human-AI approach that combines a machine learning model with crowdsourcing to evaluate ideas. Our approach models workers' reliability and bias while leveraging ideas' textual content to train a machine learning model. It is able to incorporate experts' ratings whenever available, to supervise the model training and infer worker performance. Results show that our framework outperforms baseline methods and requires significantly less training data from experts, thus providing a viable solution for evaluating ideas at scale.

Original languageEnglish
Title of host publicationACM Web Conference 2023 - Proceedings of the World Wide Web Conference, WWW 2023
PublisherAssociation for Computing Machinery (ACM)
Number of pages12
ISBN (Electronic)978-1-4503-9416-1
Publication statusPublished - 2023
Event2023 World Wide Web Conference, WWW 2023 - Austin, United States
Duration: 30 Apr 20234 May 2023

Publication series

NameACM Web Conference 2023 - Proceedings of the World Wide Web Conference, WWW 2023


Conference2023 World Wide Web Conference, WWW 2023
Country/TerritoryUnited States


  • crowdsourcing
  • human-AI collaboration
  • Idea evaluation
  • scalability


Dive into the research topics of 'HybridEval: A Human-AI Collaborative Approach for Evaluating Design Ideas at Scale'. Together they form a unique fingerprint.

Cite this