Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding

Hamdi Dibeklioglu, Zakia Hammal, Jeffrey F. Cohn

Research output: Contribution to journalArticleScientificpeer-review

63 Citations (Scopus)


Depression is one of the most common psychiatric disorders worldwide, with over 350 million people affected. Current methods to screen for and assess depression depend almost entirely on clinical interviews and self-report scales. While useful, such measures lack objective, systematic, and efficient ways of incorporating behavioral observations that are strong indicators of depression presence and severity. Using dynamics of facial and head movement and vocalization, we trained classifiers to detect three levels of depression severity. Participants were a community sample diagnosed with major depressive disorder. They were recorded in clinical interview (Hamilton Rating Scale for Depression, HRSD) at 7-week intervals over a period of 21 weeks. At each interview, they were scored by HRSD as moderately to severely depressed, mildly depressed, or remitted. Logistic regression classifiers using leave-one-participant-out validation were compared for facial movement, head movement, and vocal prosody individually and in combination. Accuracy of depression severity measurement from facial movement dynamics was higher than that for head movement dynamics; and each was substantially higher than that for vocal prosody. Accuracy using all three modalities combined only marginally exceeded that of face and head combined. These findings suggest that automatic detection of depression severity from behavioral indicators in patients is feasible and that multimodal measures afford most powerful detection.
Original languageEnglish
Pages (from-to)525-536
Number of pages12
JournalIEEE Journal of Biomedical and Health Informatics
Issue number2
Publication statusPublished - 2018


  • Depression severity
  • Facial movement dynamics
  • Head movement dynamics
  • Vocal prosody
  • Multimodal fusion


Dive into the research topics of 'Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding'. Together they form a unique fingerprint.

Cite this