TY - GEN
T1 - Value- Aware Active Learning
AU - Sayin, Burcu
AU - Yang, Jie
AU - Passerini, Andrea
AU - Casati, Fabio
PY - 2023
Y1 - 2023
N2 - In many practical applications, machine learning models are embedded into a pipeline involving a human actor that decides whether to trust the machine prediction or take a default route (e.g., classify the example herself). Selective classifiers have the option to abstain from making a prediction on an example they do not feel confident about. Recently, the notion of the value of a machine learning model has been introduced as a way to jointly consider the benefit of a correct prediction, the cost of an error, and that of abstaining. In this paper, we study how active learning of selective classifiers is affected by the focus on value. We show that the performance of the state-of-the-art active learning strategies drops significantly when we evaluate them based on value rather than accuracy. Finally, we propose a novel value-aware active learning strategy that outperforms the state-of-the-art ones when the cost of incorrect predictions substantially outweighs that of abstaining.
AB - In many practical applications, machine learning models are embedded into a pipeline involving a human actor that decides whether to trust the machine prediction or take a default route (e.g., classify the example herself). Selective classifiers have the option to abstain from making a prediction on an example they do not feel confident about. Recently, the notion of the value of a machine learning model has been introduced as a way to jointly consider the benefit of a correct prediction, the cost of an error, and that of abstaining. In this paper, we study how active learning of selective classifiers is affected by the focus on value. We show that the performance of the state-of-the-art active learning strategies drops significantly when we evaluate them based on value rather than accuracy. Finally, we propose a novel value-aware active learning strategy that outperforms the state-of-the-art ones when the cost of incorrect predictions substantially outweighs that of abstaining.
KW - active learning
KW - cost-sensitive learning
KW - selective classifier
KW - value-based learning
UR - http://www.scopus.com/inward/record.url?scp=85171455374&partnerID=8YFLogxK
U2 - 10.3233/FAIA230085
DO - 10.3233/FAIA230085
M3 - Conference contribution
AN - SCOPUS:85171455374
T3 - Frontiers in Artificial Intelligence and Applications
SP - 215
EP - 223
BT - HHAI 2023
A2 - Lukowicz, Paul
A2 - Mayer, Sven
A2 - Koch, Janin
A2 - Shawe-Taylor, John
A2 - Tiddi, Ilaria
PB - IOS Press
T2 - 2nd International Conference on Hybrid Human-Artificial Intelligence, HHAI 2023
Y2 - 26 June 2023 through 30 June 2023
ER -