Auditory Perception

Decoding speech from the brain

During speech listening, patterns of brain activity become temporally coupled to stimulus features. We can use non-invasive neuroimaging techniques, such as electroencephalography (EEG), to investigate this coupling. One exciting possibility using this method is that we can reconstruct or "decode" speech information directly from the listener's neural data. Decoding speech may hold promise as an objective measure or biomarker to diagnose and assess outcomes for people with hearing differences, including hearing loss and/or cochlear implants (CI). But the interpretation of speech decoding is not straightforward: Speech perception is a highly complex process that relies on the constant interaction between our peripheral auditory system and areas of our brain associated with so-called higher faculties, including memory and learning, for example. 

To disentangle some of the many factors involved in speech perception, this ongoing project uses data collected from 38 typically hearing listeners. They listened to a Sherlock Holmes story in a language they could understand (English), and in a similar language that they could not understand (Dutch). In addition to speech comprehension, the audio quality of the story also varied over the experiment, from natural speech to very degraded, almost robotic-sounding speech. This way, we can better understand how auditory or acoustic processing interacts with linguistic and semantic processing.

In our first paper using this data set, we found that a low-level property of sound, its intensity or loudness, is decoded with greater accuracy when the listener understands the speech they are listening to. In the next steps, we will look at other features, such as text transcriptions, to see whether language-specific features are also decodeable from the brain, and how this decoding may be affected by poor audio quality.