The NLBSE'23 Tool Competition

Rafael Kallis; Maliheh Izadi; Luca Pascarella; Oscar Chaparro; Pooja Rani

doi:10.1109/NLBSE59153.2023.00007

The NLBSE'23 Tool Competition

Rafael Kallis, Maliheh Izadi, Luca Pascarella, Oscar Chaparro, Pooja Rani

Software Engineering

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

6 Citations (Scopus)

13 Downloads (Pure)

Abstract

We report on the organization and results of the second edition of the tool competition from the International Workshop on Natural Language-based Software Engineering (NLBSE'23). As in the prior edition, we organized the competition on automated issue report classification, with a larger dataset. This year, we featured an extra competition on au-tomated code comment classification. In this tool competition edition, five teams submitted multiple classification models to automatically classify issue reports and code comments. The submitted models were fine-tuned and evaluated on a benchmark dataset of 1.4 million issue reports or 6.7 thousand code comments, respectively. The goal of the competition was to improve the classification performance of the baseline models that we provided. This paper reports details of the competition, including the rules, the teams and contestant models, and the ranking of models based on their average classification performance across issue report and code comment types.

Original language	English
Title of host publication	Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering, NLBSE 2023
Editors	Lisa O'Conner
Place of Publication	Piscataway
Publisher	IEEE
Pages	1-8
Number of pages	8
ISBN (Electronic)	979-8-3503-0178-6
ISBN (Print)	979-8-3503-0179-3
DOIs	https://doi.org/10.1109/NLBSE59153.2023.00007
Publication status	Published - 2023
Event	2nd IEEE/ACM International Workshop on Natural Language-Based Software Engineering, NLBSE 2023 - Melbourne, Australia Duration: 20 May 2023 → …

Conference

Conference	2nd IEEE/ACM International Workshop on Natural Language-Based Software Engineering, NLBSE 2023
Country/Territory	Australia
City	Melbourne
Period	20/05/23 → …

Bibliographical note

Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

Benchmark
Code Comments
Issue Reports
Labeling
Tool-Competition

Access to Document

10.1109/NLBSE59153.2023.00007

The_NLBSE23_Tool_CompetitionFinal published version, 253 KB

Cite this

@inproceedings{8fdb14ae711245fb82cee9e4ec151b77,

title = "The NLBSE'23 Tool Competition",

abstract = "We report on the organization and results of the second edition of the tool competition from the International Workshop on Natural Language-based Software Engineering (NLBSE'23). As in the prior edition, we organized the competition on automated issue report classification, with a larger dataset. This year, we featured an extra competition on au-tomated code comment classification. In this tool competition edition, five teams submitted multiple classification models to automatically classify issue reports and code comments. The submitted models were fine-tuned and evaluated on a benchmark dataset of 1.4 million issue reports or 6.7 thousand code comments, respectively. The goal of the competition was to improve the classification performance of the baseline models that we provided. This paper reports details of the competition, including the rules, the teams and contestant models, and the ranking of models based on their average classification performance across issue report and code comment types.",

keywords = "Benchmark, Code Comments, Issue Reports, Labeling, Tool-Competition",

author = "Rafael Kallis and Maliheh Izadi and Luca Pascarella and Oscar Chaparro and Pooja Rani",

note = "Green Open Access added to TU Delft Institutional Repository {\textquoteleft}You share, we take care!{\textquoteright} – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public. ; 2nd IEEE/ACM International Workshop on Natural Language-Based Software Engineering, NLBSE 2023 ; Conference date: 20-05-2023",

year = "2023",

doi = "10.1109/NLBSE59153.2023.00007",

language = "English",

isbn = "979-8-3503-0179-3",

pages = "1--8",

editor = "Lisa O'Conner",

booktitle = "Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering, NLBSE 2023",

publisher = "IEEE",

address = "United States",

}

Kallis, R, Izadi, M, Pascarella, L, Chaparro, O & Rani, P 2023, The NLBSE'23 Tool Competition. in L O'Conner (ed.), Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering, NLBSE 2023. IEEE, Piscataway, pp. 1-8, 2nd IEEE/ACM International Workshop on Natural Language-Based Software Engineering, NLBSE 2023, Melbourne, Australia, 20/05/23. https://doi.org/10.1109/NLBSE59153.2023.00007

The NLBSE'23 Tool Competition. / Kallis, Rafael; Izadi, Maliheh; Pascarella, Luca et al.
Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering, NLBSE 2023. ed. / Lisa O'Conner. Piscataway: IEEE, 2023. p. 1-8.

Research output: Chapter in Book/Conference proceedings/Edited volume › Conference contribution › Scientific › peer-review

TY - GEN

T1 - The NLBSE'23 Tool Competition

AU - Kallis, Rafael

AU - Izadi, Maliheh

AU - Pascarella, Luca

AU - Chaparro, Oscar

AU - Rani, Pooja

N1 - Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

PY - 2023

Y1 - 2023

N2 - We report on the organization and results of the second edition of the tool competition from the International Workshop on Natural Language-based Software Engineering (NLBSE'23). As in the prior edition, we organized the competition on automated issue report classification, with a larger dataset. This year, we featured an extra competition on au-tomated code comment classification. In this tool competition edition, five teams submitted multiple classification models to automatically classify issue reports and code comments. The submitted models were fine-tuned and evaluated on a benchmark dataset of 1.4 million issue reports or 6.7 thousand code comments, respectively. The goal of the competition was to improve the classification performance of the baseline models that we provided. This paper reports details of the competition, including the rules, the teams and contestant models, and the ranking of models based on their average classification performance across issue report and code comment types.

AB - We report on the organization and results of the second edition of the tool competition from the International Workshop on Natural Language-based Software Engineering (NLBSE'23). As in the prior edition, we organized the competition on automated issue report classification, with a larger dataset. This year, we featured an extra competition on au-tomated code comment classification. In this tool competition edition, five teams submitted multiple classification models to automatically classify issue reports and code comments. The submitted models were fine-tuned and evaluated on a benchmark dataset of 1.4 million issue reports or 6.7 thousand code comments, respectively. The goal of the competition was to improve the classification performance of the baseline models that we provided. This paper reports details of the competition, including the rules, the teams and contestant models, and the ranking of models based on their average classification performance across issue report and code comment types.

KW - Benchmark

KW - Code Comments

KW - Issue Reports

KW - Labeling

KW - Tool-Competition

UR - http://www.scopus.com/inward/record.url?scp=85150003471&partnerID=8YFLogxK

U2 - 10.1109/NLBSE59153.2023.00007

DO - 10.1109/NLBSE59153.2023.00007

M3 - Conference contribution

AN - SCOPUS:85150003471

SN - 979-8-3503-0179-3

SP - 1

EP - 8

BT - Proceedings of the 2023 IEEE/ACM 2nd International Workshop on Natural Language-Based Software Engineering, NLBSE 2023

A2 - O'Conner, Lisa

PB - IEEE

CY - Piscataway

T2 - 2nd IEEE/ACM International Workshop on Natural Language-Based Software Engineering, NLBSE 2023

Y2 - 20 May 2023

ER -

The NLBSE'23 Tool Competition

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this