The Shape of Learning Curves: A Review

Tom Viering*, Marco Loog

*Corresponding author for this work

Research output: Contribution to journalReview articlepeer-review

16 Citations (Scopus)
48 Downloads (Pure)

Abstract

Learning curves provide insight into the dependence of a learner's generalization performance on the training set size. This important tool can be used for model selection, to predict the effect of more training data, and to reduce the computational complexity of model training and hyperparameter tuning. This review recounts the origins of the term, provides a formal definition of the learning curve, and briefly covers basics such as its estimation. Our main contribution is a comprehensive overview of the literature regarding the shape of learning curves. We discuss empirical and theoretical evidence that supports well-behaved curves that often have the shape of a power law or an exponential. We consider the learning curves of Gaussian processes, the complex shapes they can display, and the factors influencing them. We draw specific attention to examples of learning curves that are ill-behaved, showing worse learning performance with more training data. To wrap up, we point out various open problems that warrant deeper empirical and theoretical investigation. All in all, our review underscores that learning curves are surprisingly diverse and no universal model can be identified.

Original languageEnglish
Pages (from-to)7799-7819
Number of pages21
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume45
Issue number6
DOIs
Publication statusPublished - 2023

Bibliographical note

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care
Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.

Keywords

  • Learning curve
  • training set size
  • supervised learning
  • classification
  • regression

Fingerprint

Dive into the research topics of 'The Shape of Learning Curves: A Review'. Together they form a unique fingerprint.

Cite this