Brain activity changes related to learning of audiovisual associations in reading

Abstract

Learning to connect letters or characters of written scripts to their corresponding sounds is crucial for reading acquisition. In alphabetic languages, the letter– speech sound integration has been shown to have a protracted developmental trajectory, and failure to reach an automatic level of audiovisual integration was correlated with reading difficulties. This dissertation aims to systematically investigate the audiovisual integration process in learning to read using magnetoencephalography, by extending the previous findings on alphabetic language to logographic language, and furthermore, by examining the learning of grapheme–phoneme association during initial learning stages. Study I aimed to investigate the audiovisual integration process in a logographic language (Chinese). This audiovisual integration involved the left superior temporal cortex in Chinese, which is similar to findings in alphabetic languages. In addition, it also activated the left inferior frontal regions, which are related to the processing of additional semantic information embedded in Chinese characters. Study II correlated various brain indices of audiovisual processing with reading-related cognitive measures in children at varying stages of reading. It demonstrated that the auditory late component is closely related to rapid automatized naming and phonological processing skills. Moreover, the multisensory interaction effect was observed mainly in temporoparietal regions, and brain responses in some of these regions were further associated with children’s reading and writing abilities. Study III simulated the initial learning of grapheme–phoneme associations in adults. The results from Study III highlighted the dynamic characteristics of audiovisual learning and provided a more refined model of grapheme–phoneme learning in reading acquisition. Overall, the findings from this dissertation showed evidence that audiovisual processing is dynamic during initial learning and memory consolidation of cross-modal associations. Furthermore, audiovisual processing is less automatic in children and is linked to their reading-related cognitive skills. Finally, there are some universal audiovisual processing brain regions and mechanisms across languages that are complemented by additional regions related to processes of distinct linguistic features in different types of scripts. Keywords: audiovisual integration, language learning, child brain, magnetoencephalography, reading