From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning

Jingkuan Song, Yuyu Guo, Lianli Gao, Xuelong Li, Alan Hanjalic, Heng Tao Shen

Research output: Contribution to journalArticleScientificpeer-review

85 Citations (Scopus)
104 Downloads (Pure)


Video captioning, in essential, is a complex natural process, which is affected by various uncertainties stemming from video content, subjective judgment, and so on. In this paper, we build on the recent progress in using encoder-decoder framework for video captioning and address what we find to be a critical deficiency of the existing methods that most of the decoders propagate deterministic hidden states. Such complex uncertainty cannot be modeled efficiently by the deterministic models. In this paper, we propose a generative approach, referred to as multimodal stochastic recurrent neural networks (MS-RNNs), which models the uncertainty observed in the data using latent stochastic variables. Therefore, MS-RNN can improve the performance of video captioning and generate multiple sentences to describe a video considering different random factors. Specifically, a multimodal long short-term memory (LSTM) is first proposed to interact with both visual and textual features to capture a high-level representation. Then, a backward stochastic LSTM is proposed to support uncertainty propagation by introducing latent variables. Experimental results on the challenging data sets, microsoft video description and microsoft research video-to-text, show that our proposed MS-RNN approach outperforms the state-of-the-art video captioning benchmarks.

Original languageEnglish
Pages (from-to)1-12
Number of pages12
JournalIEEE Transactions on Neural Networks and Learning Systems
Publication statusE-pub ahead of print - 2018

Bibliographical note

Accepted Author Manuscript


  • Recurrent neural network (RNN)
  • uncertainty
  • video captioning.


Dive into the research topics of 'From Deterministic to Generative: Multimodal Stochastic RNNs for Video Captioning'. Together they form a unique fingerprint.

Cite this