Understanding Human Manipulation with the Environment: A Novel Taxonomy for Video Labelling

Visar Arapi*, Cosimo Della Santina, Giuseppe Averta, Antonio Bicchi, Matteo Bianchi

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

10 Downloads (Pure)


In recent years, the spread of data-driven approaches for robotic grasp synthesis has come with the increasing need for reliable datasets, which can be built e.g. through video labelling. To this goal, it is important to define suitable rules to characterize the main human grasp types, for easily identifying them in video streams. In this work, we present a novel taxonomy that builds upon the related state of the art, but it is specifically thought for video labelling. It focuses on the interaction of the hand with the environment and accounts for pre-contact phases, bi-manual grasps as well as non-prehensile strategies. This study is complemented with a dataset of labelled videos of subjects performing activities of daily living, for a total of nine hours, and the description of MatLab tools for labelling new videos. Both hands were labelled at any time. We used these labelled data for performing a preliminary statistical description of the occurrences of the here proposed class types.

Original languageEnglish
Pages (from-to)6537-6544
JournalIEEE Robotics and Automation Letters
Issue number4
Publication statusPublished - 2021

Bibliographical note

Green Open Access added to TU Delft Institutional Repository 'You share, we take care!' - Taverne project https://www.openaccess.nl/en/you-share-we-take-care

Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.


  • bimanual manipulation
  • Datasets for human motion
  • deep learning
  • dexterous manipulation


Dive into the research topics of 'Understanding Human Manipulation with the Environment: A Novel Taxonomy for Video Labelling'. Together they form a unique fingerprint.

Cite this