CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences

Maliheh Izadi, Roberta Gismondi, Georgios Gousios

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

11 Citations (Scopus)
84 Downloads (Pure)


Code completion is an essential feature of IDEs, yet current auto-completers are restricted to either grammar-based or NLP-based single token completions. Both approaches have significant draw-backs: grammar-based autocompletion is restricted in dynamically-typed language environments, whereas NLP-based autocompleters struggle to understand the semantics of the programming language and the developer's code context. In this work, we present CodeFill, a language model for autocompletion that combines learned structure and naming information. Using a parallel Transformer architecture and multi-task learning, CodeFill consumes sequences of source code token names and their equivalent AST token types. Uniquely, CodeFill is trained both for single-token and multi-token (statement) prediction, which enables it to learn long-range dependencies among grammatical and naming elements. We train CodeFill on two datasets, consisting of 29M and 425M lines of code, respectively. To make the evaluation more realistic, we develop a method to automatically infer points in the source code at which completion matters. We compare CodeFill against four baselines and two state-of-the-art models, GPT-C and TravTrans+. CodeFill surpasses all baselines in single token prediction (MRR: 70.9% vs. 66.2% and 67.8%) and outperforms the state of the art for multi-token prediction (ROUGE-L: 63.7% vs. 52.4% and 59.2%, for n=4 tokens). We publicly release our source code and datasets.

Original languageEnglish
Title of host publicationProceedings - 2022 ACM/IEEE 44th International Conference on Software Engineering, ICSE 2022
Number of pages12
ISBN (Electronic)9781450392211
Publication statusPublished - 2022
Event44th ACM/IEEE International Conference on Software Engineering, ICSE 2022: Software Engineering in Practice (ICSE-SEIP) - Pittsburgh, United States
Duration: 22 May 202227 May 2022
Conference number: 44th

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257


Conference44th ACM/IEEE International Conference on Software Engineering, ICSE 2022
Abbreviated title ICSE 2022
Country/TerritoryUnited States


  • Automatic Code Completion
  • Dynamically-typed Languages
  • Multi-Task Learning
  • Transformers
  • Types


Dive into the research topics of 'CodeFill: Multi-token Code Completion by Jointly learning from Structure and Naming Sequences'. Together they form a unique fingerprint.

Cite this