TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools

Ashwin Prasad S. Venkatesh, Samkutty Sabu, Jiawei Wang, Amir M. Mir, Li Li, Eric Bodden

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

10 Downloads (Pure)

Abstract

In light of the growing interest in type inference research for Python, both researchers and practitioners require a standardized process to assess the performance of various type inference techniques. This paper introduces TypeEvalPy, a comprehensive microbenchmarking framework for evaluating type inference tools. Type- EvalPy contains 154 code snippets with 845 type annotations across 18 categories that target various Python features. The framework manages the execution of containerized tools, transforms inferred types into a standardized format, and produces meaningful metrics for assessment. Through our analysis, we compare the performance of six type inference tools, highlighting their strengths and limitations. Our findings provide a foundation for further research and optimization in the domain of Python type inference.

Original languageEnglish
Title of host publicationProceedings - 2024 ACM/IEEE 46th International Conference on Software Engineering
Subtitle of host publicationCompanion, ICSE-Companion 2024
PublisherIEEE
Pages49-53
Number of pages5
ISBN (Electronic)9798400705021
DOIs
Publication statusPublished - 2024
Event46th International Conference on Software Engineering: Companion, ICSE-Companion 2024 - Lisbon, Portugal
Duration: 14 Apr 202420 Apr 2024

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257

Conference

Conference46th International Conference on Software Engineering: Companion, ICSE-Companion 2024
Country/TerritoryPortugal
CityLisbon
Period14/04/2420/04/24

Fingerprint

Dive into the research topics of 'TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools'. Together they form a unique fingerprint.

Cite this