Interpretable embeddings of speech explain and enhance the brain encoding performance of audio models.

In ECoG recordings, speech foundation models’ brain alignment is largely explained by linear encoding using interpretable speech feature families, with variance partitioning across feature classes.

• Why it matters: The field lacks a clear account of what in speech foundation model representations matches brain responses, limiting interpretability and the design of better brain-encoding models.
• What they did: The authors fit linear encoding models using six interpretable feature families plus contextualized embeddings from Whisper, HuBERT, and WavLM, then partition shared ECoG response variance across feature classes and layers.
• The result: Alignment is mostly driven by simple interpretable speech features; layers show a trade-off between low- and high-level encoding, and higher-level semantics linked to brain responses increases with model size and context length.

AI summary of the abstract · may contain errors · not medical advice · read the paper.