On the Evaluation of NLP-based Models for Software Engineering

Maliheh Izadi, Martin Nili Ahmadabadi

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

4 Citations (Scopus)
66 Downloads (Pure)

Abstract

NLP-based models have been increasingly incorporated to address SE problems. These models are either employed in the SE domain with little to no change, or they are greatly tailored to source code and its unique characteristics. Many of these approaches are considered to be outperforming or complementing existing solutions. However, an important question arises here: Are these models evaluated fairly and consistently in the SE community?. To answer this question, we reviewed how NLP-based models for SE problems are being evaluated by researchers. The findings indicate that currently there is no consistent and widely-accepted protocol for the evaluation of these models. While different aspects of the same task are being assessed in different studies, metrics are defined based on custom choices, rather than a system, and finally, answers are collected and interpreted case by case. Consequently, there is a dire need to provide a methodological way of evaluating NLP-based models to have a consistent assessment and preserve the possibility of fair and efficient comparison.
Original languageEnglish
Title of host publicationProceedings of the 2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE)
PublisherIEEE
Pages48-50
Number of pages3
ISBN (Electronic)978-1-4503-9343-0
ISBN (Print)978-1-6654-6231-0
DOIs
Publication statusPublished - 2022
Event2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE) - Pittsburgh, United States
Duration: 8 May 20228 May 2022

Publication series

NameProceedings - 1st International Workshop on Natural Language-Based Software Engineering, NLBSE 2022

Workshop

Workshop2022 IEEE/ACM 1st International Workshop on Natural Language-Based Software Engineering (NLBSE)
Country/TerritoryUnited States
CityPittsburgh
Period8/05/228/05/22

Keywords

  • Evaluation
  • Natural Language Processing
  • Software Engineering

Fingerprint

Dive into the research topics of 'On the Evaluation of NLP-based Models for Software Engineering'. Together they form a unique fingerprint.

Cite this