TY - GEN
T1 - What do You Mean? Interpreting Image Classification with Crowdsourced Concept Extraction and Analysis
AU - Balayn, Agathe
AU - Soilis, Panagiotis
AU - Lofi, Christoph
AU - Yang, Jie
AU - Bozzon, Alessandro
PY - 2021
Y1 - 2021
N2 - Global interpretability is a vital requirement for image classification applications. Existing interpretability methods mainly explain a model behavior by identifying salient image patches, which require manual efforts from users to make sense of, and also do not typically support model validation with questions that investigate multiple visual concepts. In this paper, we introduce a scalable human-in-the-loop approach for global interpretability. Salient image areas identified by local interpretability methods are annotated with semantic concepts, which are then aggregated into a tabular representation of images to facilitate automatic statistical analysis of model behavior. We show that this approach answers interpretability needs for both model validation and exploration, and provides semantically more diverse, informative, and relevant explanations while still allowing for scalable and cost-efficient execution.
AB - Global interpretability is a vital requirement for image classification applications. Existing interpretability methods mainly explain a model behavior by identifying salient image patches, which require manual efforts from users to make sense of, and also do not typically support model validation with questions that investigate multiple visual concepts. In this paper, we introduce a scalable human-in-the-loop approach for global interpretability. Salient image areas identified by local interpretability methods are annotated with semantic concepts, which are then aggregated into a tabular representation of images to facilitate automatic statistical analysis of model behavior. We show that this approach answers interpretability needs for both model validation and exploration, and provides semantically more diverse, informative, and relevant explanations while still allowing for scalable and cost-efficient execution.
KW - Concept extraction
KW - Human computation
KW - Image classification
KW - Machine-learning interpretability
UR - http://www.scopus.com/inward/record.url?scp=85107912985&partnerID=8YFLogxK
U2 - 10.1145/3442381.3450069
DO - 10.1145/3442381.3450069
M3 - Conference contribution
SN - 978-1-4503-8312-7
T3 - The Web Conference 2021 - Proceedings of the World Wide Web Conference, WWW 2021
SP - 1937
EP - 1948
BT - The Web Conference 2021 - Proceedings of the World Wide Web Conference, WWW 2021
T2 - International World Wide Web Conference 2021
Y2 - 19 April 2021 through 23 April 2021
ER -