FMCW Radar-Based Hand Gesture Recognition using Spatiotemporal Deformable and Context-Aware Convolutional 5D Feature Representation

Xichao Dong, Zewei Zhao, Yupei Wang, Tao Zeng, Jianping Wang, Yi Sui

Research output: Contribution to journalArticleScientificpeer-review


Recently, frequency-modulated continuous wave (FMCW) radar-based hand gesture recognition using deep learning has achieved favorable performance. However, many existing methods use extracted features separately, i.e., using one of the range, Doppler, azimuth or elevation angle information, or a combination of any two, to train convolutional neural networks (CNNs), which ignore the interrelation among the 5D time-varying-range-Doppler-azimuth-elevation feature space. Although there have been methods using the 5D information, their mining of the interrelation among the 5D feature space is not sufficient, and there’s still room for improvements. This paper proposes a new processing scheme of hand gesture recognition based on 5D feature cubes which are jointly encoded by a 3D fast Fourier transform (3D-FFT) based method. Then a CNN is proposed by building two novel blocks, i.e., spatiotemporal deformable convolution (STDC) block and adaptive spatiotemporal context-aware convolution (ASTCAC) block. Concretely, STDC is designed to cope with hand gestures’ large spatiotemporal geometric transformations in the 5D feature space. Moreover, ASTCAC is designed for modeling long-distance global relationships, e.g., relationships between pixels of the feature at upper left corner and lower right corner, and exploring the global spatiotemporal context, in order to enhance the target feature representation and suppress interference. Finally, our presented method is verified on a large radar dataset including 19760 sets of 16 common hand gestures, collected by 19 subjects. Our method obtains a recognition rate of 99.53% on validation dataset, and that of 97.22% on test dataset, which is significantly better than state-of-the-art methods.

Original languageEnglish
JournalIEEE Transactions on Geoscience and Remote Sensing
Publication statusPublished - 2021


  • Azimuth
  • Convolution
  • Doppler effect
  • Estimation
  • Feature extraction
  • Frequency-modulated continuous wave (FMCW) radar
  • hand gesture recognition
  • spatiotemporal context modeling
  • spatiotemporal deformable convolution
  • Spatiotemporal phenomena
  • Three-dimensional displays


Dive into the research topics of 'FMCW Radar-Based Hand Gesture Recognition using Spatiotemporal Deformable and Context-Aware Convolutional 5D Feature Representation'. Together they form a unique fingerprint.

Cite this