Parallel Real-Time Tracking and 3D Reconstruction with TBB for Intelligent Control and Smart Sensing Framework

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

5 Citations (Scopus)
166 Downloads (Pure)


Recent advancements in aircraft controllers paired with increasingly flexible aircraft designs create the need for adaptive and intelligent control systems. To correctly capture the motion of a flexible aircraft wing and provide feedback to the controller, a large number of states (nodes along the span) must be monitored in real-time. Visual sensing methods carry the promise of flexibility needed for this type of smart sensing and control. However, visual sensing requires capturing and tracking keypoint features (marker tracking), while detecting thereof from a feature-rich image can be a computationally intensive task. The computational effort significantly increases with image size or when an image stereo pair is used to find matching keypoints. In this study, a parallel approach is presented with Threading Building Blocks (TBB), using sub-matrix computations, for extraction of corresponding keypoints from an image-stereo pair, and triangulation with the Direct Linear Transform (DLT) method to reconstruct the 3D position of the object in space. Additional robustness is investigated by implementing a Kalman filter for tracking prediction during the domain transition between the sub-matrices. Furthermore, a flexible simulation framework is set up for smart sensing with a coupled unsteady aeroservoelastic model of a 3D wing and a visual model to test the method for intelligent control feedback in a simulation environment. The methodology is tested in a laboratory environment with a stereo camera setup, and in a virtual environment, where the virtual camera parameters are reconstructed to meet a stereo setup. The proposed approach aims to advance the state-of-the-art in smart sensing, particularly in the context of real-time state estimation of aeroelastic structures and control feedback. The parallel approach shows a significant improvement of speed and efficiency, allowing real-time computation from a live image stream at 50 fps.

Original languageEnglish
Title of host publicationAIAA Scitech 2020 Forum
Subtitle of host publication6-10 January 2020, Orlando, FL
PublisherAmerican Institute of Aeronautics and Astronautics Inc. (AIAA)
Number of pages23
ISBN (Electronic)978-1-62410-595-1
Publication statusPublished - 2020
EventAIAA Scitech 2020 Forum - Orlando, United States
Duration: 6 Jan 202010 Jan 2020

Publication series

NameAIAA Scitech 2020 Forum
Volume1 PartF


ConferenceAIAA Scitech 2020 Forum
Country/TerritoryUnited States


Dive into the research topics of 'Parallel Real-Time Tracking and 3D Reconstruction with TBB for Intelligent Control and Smart Sensing Framework'. Together they form a unique fingerprint.

Cite this