NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: Are these models evaluated fairly and consistently in the SE community?. To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. While different aspects of the same task are being assessed in different studies, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there is a dire need to provide a methodological way of evaluating NLP-based models to have a consistent assessment and preserve the possibility of fair and efficient comparison.
|Title of host publication||Proceedings of the 2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE)|
|Number of pages||3|
|Publication status||Published - 2022|
|Event||2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE) - Pittsburgh, United States|
Duration: 8 May 2022 → 8 May 2022
|Name||Proceedings - 1st International Workshop on Natural Language-Based Software Engineering, NLBSE 2022|
|Workshop||2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE)|
|Period||8/05/22 → 8/05/22|
Bibliographical noteGreen Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.
- Natural Language Processing
- Software Engineering