Online function minimization with convex random relu expansions

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

Abstract

We propose CDONE, a convex version of the DONE algorithm. DONE is a derivative-free online optimization algorithm that uses surrogate modeling with noisy measurements to find a minimum of objective functions that are expensive to evaluate. Inspired by their success in deep learning, CDONE makes use of rectified linear units, together with a nonnegativity constraint to enforce convexity of the surrogate model. This leads to a sparse and cheap to evaluate surrogate model of the unknown optimization objective that is still accurate and that can be minimized with convex optimization algorithms. The CDONE algorithm is demonstrated on a toy example and on the problem of hyper-parameter optimization for a deep learning example on handwritten digit classification.

Original languageEnglish
Title of host publicationProceedings 2017 27th IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2017)
EditorsN. Ueda, T. Matsui
Place of PublicationPiscataway, NJ, USA
PublisherIEEE
Number of pages6
ISBN (Electronic)9781509063413
DOIs
Publication statusPublished - 2017
Event2017 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017 - Tokyo, Japan
Duration: 25 Sep 201728 Sep 2017

Conference

Conference2017 IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2017
CountryJapan
CityTokyo
Period25/09/1728/09/17

Keywords

  • Bayesian optimization
  • Deep learning
  • Derivative-free optimization
  • Surrogate modeling

Fingerprint Dive into the research topics of 'Online function minimization with convex random relu expansions'. Together they form a unique fingerprint.

  • Cite this

    Bliek, L., Verhaegen, M., & Wahls, S. (2017). Online function minimization with convex random relu expansions. In N. Ueda, & T. Matsui (Eds.), Proceedings 2017 27th IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2017) IEEE. https://doi.org/10.1109/MLSP.2017.8168109