The effectiveness of unsupervised subword modeling with autoregressive and cross-lingual phone-aware networks

Siyuan Feng, Odette Scharenborg

Research output: Contribution to journalArticleScientificpeer-review

13 Downloads (Pure)


This study addresses unsupervised subword modeling, i.e., learning acoustic feature representations that can distinguish between subword units of a language. We propose a two-stage learning framework that combines self-supervised learning and cross-lingual knowledge transfer. The framework consists of autoregressive predictive coding (APC) as the front-end and a cross-lingual deep neural network (DNN) as the back-end. Experiments on the ABX subword discriminability task conducted with the Libri-light and ZeroSpeech 2017 databases showed that our approach is competitive or superior to state-of-the-art studies. Comprehensive and systematic analyses at the phoneme- and articulatory feature (AF)-level showed that our approach was better at capturing diphthong than monophthong vowel information, while also differences in the amount of information captured for different types of consonants were observed. Moreover, a positive correlation was found between the effectiveness of the back-end in capturing a phoneme's information and the quality of the cross-lingual phone labels assigned to the phoneme. The AF-level analysis together with t-SNE visualization results showed that the proposed approach is better than MFCC and APC features in capturing manner and place of articulation information, vowel height, and backness information. Taken together, the analyses showed that the two stages in our approach are both effective in capturing phoneme and AF information. Nevertheless, monophthong vowel information is less well captured than consonant information, which suggests that future research should focus on improving capturing monophthong vowel information.
Original languageEnglish
Article number9420327
Pages (from-to)230 - 247
Number of pages18
JournalIEEE Open Journal of Signal Processing
Publication statusPublished - 2021


  • Unsupervised subword modeling
  • zero-resource
  • cross-lingual modeling
  • phoneme analysis
  • articulatory feature analysis


Dive into the research topics of 'The effectiveness of unsupervised subword modeling with autoregressive and cross-lingual phone-aware networks'. Together they form a unique fingerprint.

Cite this