Structured Probabilistic End-to-End Learning from Crowds

Zhijun Chen, Huimin Wang, Hailong Sun, Pengpeng Chen, Tao Han, Xudong Liu, Jie Yang

Research output: Chapter in Book/Conference proceedings/Edited volumeChapterScientificpeer-review


End-to-end learning from crowds has recently been introduced as an EM-free approach to training deep neural networks directly from noisy crowdsourced annotations. It models the relationship between true labels and annotations with a specific type of neural layer, termed as the crowd layer, which can be trained using pure backpropagation. Parameters of the crowd layer, however, can hardly be interpreted as annotator reliability, as compared with the more principled probabilistic approach. The lack of probabilistic interpretation further prevents extensions of the approach to account for important factors of annotation processes, e.g., instance difficulty. This paper presents SpeeLFC, a structured probabilistic model that incorporates the constraints of probability axioms for parameters of the crowd layer, which allows to explicitly model annotator reliability while benefiting from the end-to-end training of neural networks. Moreover, we propose SpeeLFC-D, which further takes into account instance difficulty. Extensive validation on real-world datasets shows that our methods improve the state-of-the-art.
Original languageEnglish
Title of host publicationIJCAI-20
Subtitle of host publicationProceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence
PublisherInternational Joint Conferences on Artifical Intelligence (IJCAI)
Pages1512 - 1518
Number of pages7
ISBN (Electronic)978-0-9992411-6-5
Publication statusPublished - 2020
EventThe Twenty-Ninth International Joint Conference on Artificial Intelligence - Yokohoma, Japan
Duration: 7 Jan 202115 Jan 2021
Conference number: 29th


ConferenceThe Twenty-Ninth International Joint Conference on Artificial Intelligence
Internet address

Bibliographical note

Virtual / Online event was rescheduled (2020) due to COVID-19


Dive into the research topics of 'Structured Probabilistic End-to-End Learning from Crowds'. Together they form a unique fingerprint.

Cite this