Effective crowdsourced generation of training data for chatbots natural language understanding

Rucha Bapat, Pavel Kucherbaev*, Alessandro Bozzon

*Corresponding author for this work

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

6 Citations (Scopus)
211 Downloads (Pure)


Chatbots are text-based conversational agents. Natural Language Understanding (NLU) models are used to extract meaning and intention from user messages sent to chatbots. The user experience of chatbots largely depends on the performance of the NLU model, which itself largely depends on the initial dataset the model is trained with. The training data should cover the diversity of real user requests the chatbot will receive. Obtaining such data is a challenging task even for big corporations. We introduce a generic approach to generate training data with the help of crowd workers, we discuss the approach workflow and the design of crowdsourcing tasks assuring high quality. We evaluate the approach by running an experiment collecting data for 9 different intents. We use the collected training data to train a natural language understanding model. We analyse the performance of the model under different training set sizes for each intent. We provide recommendations on selecting an optimal confidence threshold for predicting intents, based on the cost model of incorrect and unknown predictions.
Original languageEnglish
Title of host publicationWeb Engineering - 18th International Conference, ICWE 2018, Proceedings
Number of pages15
ISBN (Electronic)978-3-319-91662-0
ISBN (Print)978-3-319-91661-3
Publication statusPublished - 2018
Event18th International Conference on Web Engineering, ICWE 2018 - Caceres, Spain
Duration: 5 Jun 20188 Jun 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10845 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference18th International Conference on Web Engineering, ICWE 2018

Bibliographical note

Accepted Author Manuscript


  • Conversational agents
  • Crowdsourcing
  • Natural language understanding


Dive into the research topics of 'Effective crowdsourced generation of training data for chatbots natural language understanding'. Together they form a unique fingerprint.

Cite this