Abstract
Large Language Models (LLMs) have gained considerable traction within the Software Engineering (SE) community, impacting various SE tasks from code completion to test generation, from program repair to code summarization. Despite their promise, researchers must still be careful as numerous intricate factors can influence the outcomes of experiments involving LLMs.
This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings.
In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns.
The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.
This paper initiates an open discussion on potential threats to the validity of LLM-based research including issues such as closed-source models, possible data leakage between LLM training data and research evaluation, and the reproducibility of LLM-based findings.
In response, this paper proposes a set of guidelines tailored for SE researchers and Language Model (LM) providers to mitigate these concerns.
The implications of the guidelines are illustrated using existing good practices followed by LLM providers and a practical example for SE researchers in the context of test case generation.
Original language | English |
---|---|
Title of host publication | Proceedings - 2024 ACM/IEEE 46th International Conference on Software Engineering |
Subtitle of host publication | New Ideas and Emerging Results, ICSE-NIER 2024 |
Publisher | IEEE / ACM |
Pages | 102-106 |
Number of pages | 5 |
ISBN (Electronic) | 9798400705007 |
DOIs | |
Publication status | Published - 2024 |
Event | ACM/IEEE 46th International Conference on Software Engineering - Lisbon, Lisbon, Portugal Duration: 14 Apr 2024 → 20 Apr 2024 Conference number: 46 https://conf.researchr.org/home/icse-2024 |
Publication series
Name | Proceedings - International Conference on Software Engineering |
---|---|
ISSN (Print) | 0270-5257 |
Conference
Conference | ACM/IEEE 46th International Conference on Software Engineering |
---|---|
Abbreviated title | ICSE '24 |
Country/Territory | Portugal |
City | Lisbon |
Period | 14/04/24 → 20/04/24 |
Internet address |
Keywords
- Large Language Models
- Artificial Intelligence
- Empirical Software Engineering
- Empirical Software Validation