No frame left behind: Full Video Action Recognition

Xin Liu, Silvia L. Pintea, Fatemeh Karimi Nejadasl, Olaf Booij, Jan C. van Gemert

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

17 Citations (Scopus)


Not all video frames are equally informative for recognizing an action. It is computationally infeasible to train deep networks on all video frames when actions develop over hundreds of frames. A common heuristic is uniformly sampling a small number of video frames and using these to recognize the action. Instead, here we propose full video action recognition and consider all video frames. To make this computational tractable, we first cluster all frame activations along the temporal dimension based on their similarity with respect to the classification task, and then temporally aggregate the frames in the clusters into a smaller number of representations. Our method is end-to-end trainable and computationally efficient as it relies on temporally localized clustering in combination with fast Hamming distances in feature space. We evaluate on UCF101, HMDB51, Breakfast, and Something-Something V1 and V2, where we compare favorably to existing heuristic frame sampling methods.
Original languageEnglish
Title of host publication2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Subtitle of host publicationProceedings
EditorsL. O'Conner
Place of PublicationPiscataway
Number of pages10
ISBN (Electronic)978-1-6654-4509-2
ISBN (Print)978-1-6654-4510-8
Publication statusPublished - 2021
Event2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition
- Virtual at Nashville, United States
Duration: 20 Jun 202125 Jun 2021


Conference2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition
Abbreviated titleCVPR 2021
Country/TerritoryUnited States
CityVirtual at Nashville


Dive into the research topics of 'No frame left behind: Full Video Action Recognition'. Together they form a unique fingerprint.

Cite this