Visualizing Phoneme Category Adaptation in Deep Neural Networks

Odette Scharenborg, Sebastian Tiesmeyer, Mark Hasegawa-Johnson, Najim Dehak

Research output: Chapter in Book/Conference proceedings/Edited volumeConference contributionScientificpeer-review

7 Citations (Scopus)
37 Downloads (Pure)


Both human listeners and machines need to adapt their sound categories whenever a new speaker is encountered. This perceptual learning is driven by lexical information. The aim of this paper is two-fold: investigate whether a deep neural network-based (DNN) ASR system can adapt to only a few examples of ambiguous speech as humans have been found to do; investigate a DNN’s ability to serve as a model of human perceptual learning. Crucially, we do so by looking at intermediate levels of phoneme category adaptation rather than at the output level. We visualize the activations in the hidden layers of the DNN during perceptual learning. The results show that, similar to humans, DNN systems learn speaker-adapted phone category boundaries from a few labeled examples. The DNN adapts its category boundaries not only by adapting the weights of the output layer, but also by adapting the implicit feature maps computed by the hidden layers, suggesting the possibility that human perceptual learning might involve a similar nonlinear distortion of a perceptual space that is intermediate between the acoustic input and the phonological categories. Comparisons between DNNs and humans can thus provide valuable insights into the way humans process speech and improve ASR technology.
Original languageEnglish
Title of host publicationProceedings of Interspeech 2018
EditorsB. Yegnanarayana
Place of PublicationIndia
PublisherInternational Speech Communication Association
Number of pages5
Publication statusPublished - 3 Sept 2018
EventInterspeech 2018 - Hyderabad, India
Duration: 2 Sept 20186 Sept 2018


ConferenceInterspeech 2018


  • phoneme category adaptation
  • human perceptual learning
  • deep neural networks
  • Visualisation


Dive into the research topics of 'Visualizing Phoneme Category Adaptation in Deep Neural Networks'. Together they form a unique fingerprint.

Cite this