Reconstruction of Pilot Behaviour from Cockpit Image Recorder

Hiroka Tsuda, Olaf Stroosma, Max Mulder

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

70 Downloads (Pure)


A method to automatically identify pilot actions from cockpit camera footage is reported in this paper. Although they have long been considered for the enhancement of flight safety, cockpit image recorders have not yet been standard equipment in aircraft cockpits. The rules on Flight Data Recorders have been changed, however, to include a cockpit image recorder as one of the safety devices, and it is recommended to be installed in small aircraft as a substitute for a Flight Data Recorder. With cockpit images becoming available, it would surely be useful for accident analysis as well as for daily flight analysis. Especially for the latter purpose, pilot behavior should be automatically analyzed and classified into specific actions, or procedures. The authors conducted a study to assess the feasibility of automatic detection of pilot actions in the cockpit by a machine learning process. Results show that even with a small amount of training data, the resulting algorithm could identify some typical actions, such as manipulation of the switches on the glare shield, with 80% accuracy. Even in cases with a button and a switch positioned very close to each other, the actions ‘pushing the switch’ and ‘pushing the button’ could be distinguished by the algorithm. The action estimation accuracy improves up to 90% when using the training data focused on the pilot’s body parts, rather than the data focused on the whole body.

Original languageEnglish
Title of host publicationAIAA Scitech 2020 Forum
Subtitle of host publication6-10 January 2020, Orlando, FL
PublisherAmerican Institute of Aeronautics and Astronautics Inc. (AIAA)
Number of pages9
ISBN (Electronic)978-1-62410-595-1
Publication statusPublished - 2020
EventAIAA Scitech 2020 Forum - Orlando, United States
Duration: 6 Jan 202010 Jan 2020

Publication series

NameAIAA Scitech 2020 Forum
Volume1 PartF


ConferenceAIAA Scitech 2020 Forum
CountryUnited States

Bibliographical note

Invited paper


Dive into the research topics of 'Reconstruction of Pilot Behaviour from Cockpit Image Recorder'. Together they form a unique fingerprint.

Cite this