Active vision via extremum seeking for robots in unstructured environments: Applications in object recognition and manipulation

Berk Calli, Wouter Caarls, Martijn Wisse, Pieter P. Jonker

Research output: Contribution to journalArticleScientificpeer-review

18 Citations (Scopus)
194 Downloads (Pure)


In this paper, a novel active vision strategy is proposed for optimizing the viewpoint of a robot's vision sensor for a given success criterion. The strategy is based on extremum seeking control (ESC), which introduces two main advantages: 1) Our approach is model free: It does not require an explicit objective function or any other task model to calculate the gradient direction for viewpoint optimization. This brings new possibilities for the use of active vision in unstructured environments, since a priori knowledge of the surroundings and the target objects is not required. 2) ESC conducts continuous optimization backed up with mechanisms to escape from local maxima. This enables an efficient execution of an active vision task. We demonstrate our approach with two applications in the object recognition and manipulation fields, where the model-free approach brings various benefits: for object recognition, our framework removes the dependence on offline training data for viewpoint optimization, and provides robustness of the system to occlusions and changing lighting conditions. In object manipulation, the model-free approach allows us to increase the success rate of a grasp synthesis algorithm without the need of an object model; the algorithm only uses continuous measurements of the objective value, i.e., the grasp quality. Our experiments show that continuous viewpoint optimization can efficiently increase the data quality for the underlying algorithm, while maintaining the robustness.

Original languageEnglish
Pages (from-to)1810-1822
JournalIEEE Transactions on Automation Science and Engineering
Issue number4
Publication statusPublished - 2018

Bibliographical note

Accepted Author Manuscript


  • Active vision
  • Artificial neural networks
  • extremum seeking control (ESC)
  • grasping
  • manipulation.
  • Object recognition
  • object recognition
  • Optimization
  • Robot sensing systems
  • Robustness
  • Task analysis


Dive into the research topics of 'Active vision via extremum seeking for robots in unstructured environments: Applications in object recognition and manipulation'. Together they form a unique fingerprint.

Cite this