Heed the noise in performance evaluations in neural architecture search

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

19 Downloads (Pure)

Abstract

Neural Architecture Search (NAS) has recently become a topic of great interest. However, there is a potentially impactful issue within NAS that remains largely unrecognized: noise. Due to stochastic factors in neural network initialization, training, and the chosen train/validation dataset split, the performance evaluation of a neural network architecture, which is often based on a single learning run, is also stochastic. This may have a particularly large impact if a dataset is small. We therefore propose to reduce this noise by evaluating architectures based on average performance over multiple network training runs using different random seeds and cross-validation. We perform experiments for a combinatorial optimization formulation of NAS in which we vary noise reduction levels. We use the same computational budget for each noise level in terms of network training runs, i.e., we allow less architecture evaluations when averaging over more training runs. Multiple search algorithms are considered, including evolutionary algorithms which generally perform well for NAS. We use two publicly available datasets from the medical image segmentation domain where datasets are often limited and variability among samples is often high. Our results show that reducing noise in architecture evaluations enables finding better architectures by all considered search algorithms.

Original languageEnglish
Title of host publicationGECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference
PublisherAssociation for Computing Machinery (ACM)
Pages2104-2112
Number of pages9
ISBN (Electronic)978-1-4503-9268-6
DOIs
Publication statusPublished - 2022
Event2022 Genetic and Evolutionary Computation Conference, GECCO 2022 - Virtual, Online, United States
Duration: 9 Jul 202213 Jul 2022

Publication series

NameGECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference

Conference

Conference2022 Genetic and Evolutionary Computation Conference, GECCO 2022
Country/TerritoryUnited States
CityVirtual, Online
Period9/07/2213/07/22

Keywords

  • medical image segmentation
  • neural architecture search
  • noise

Fingerprint

Dive into the research topics of 'Heed the noise in performance evaluations in neural architecture search'. Together they form a unique fingerprint.

Cite this