Developing Automated Feedback on Spoken Performance : Exploring the Functioning of Five Analytic Rating Scales Using Many-facet Rasch Measurement
Zansen, A. V., & Huhta, A. (2022). Developing Automated Feedback on Spoken Performance : Exploring the Functioning of Five Analytic Rating Scales Using Many-facet Rasch Measurement . In J. H. Jantunen, J. Kalja-Voima, M. Laukkarinen, A. Puupponen, M. Salonen, T. Saresma, J. Tarvainen, & S. Ylönen (Eds.), Diversity of Methods and Materials in Digital Human Sciences : Proceedings of the Digital Research Data and Human Sciences DRDHum Conference 2022, December 1-3, Jyväskylä, Finland (pp. 211-229). Jyväskylän yliopisto. http://urn.fi/URN:ISBN:978-951-39-9450-1
© 2022 Authors and University of Jyväskylä
In this study, we used the Many-facet Rasch measurement (MFRM) to explore the quality of ratings as well as the functioning of five analytic rating scales developed for automated assessment of L2 speech. This study is part of a multidisciplinary research project that develops automatic speech recognition (ASR), automated scoring and automated feedback for L2 Finnish and Swedish. The data include the analytic ratings (task completion, fluency, pronunciation, range, accuracy) gathered from human raters (n=14) who assessed L2 Finnish learners’ (n=64) speech samples using Moodle. The four-facet Rasch analysis showed that the raters performed and the rating scales functioned well, although task completion seems to be more challenging to apply consistently than the other criteria. Moreover, it proved to be more difficult to receive a certain score on some dimensions, namely fluency and range, than others. The study has implications for score reporting. We demonstrated that a) the different analytical rating scales have somewhat different structure, b) scores do not advance with equal intervals and c) a certain score on a certain dimension might require a bigger leap forward in ability than on other dimensions. The results will be used for designing encouraging and accurate automated feedback to L2 Finnish and Swedish learners. ...
Parent publication ISBN978-951-39-9450-1
ConferenceDigital Research Data and Human Sciences
Is part of publicationDiversity of Methods and Materials in Digital Human Sciences : Proceedings of the Digital Research Data and Human Sciences DRDHum Conference 2022, December 1-3, Jyväskylä, Finland
Publication in research information system
MetadataShow full item record
Related funder(s)Academy of Finland
Funding program(s)Academy Project, AoF
Additional information about fundingThe DigiTala project is funded by the Academy of Finland 2019–2023, and combines expertise in speech and language processing, language education and phonetics at the University of Helsinki (grant number 322619), Aalto University (grant number 322625) and the University of Jyväskylä (grant number 322965).
Showing items with similar title or keywords.
Prosody and fluency of Finland Swedish as a second language : Investigating global parameters for automated speaking assessment Kallio, Heini; Kautonen, Maria; Kuronen, Mikko (Elsevier BV, 2023)This study investigates prosody and fluency of Finland Swedish as a second language (L2). The main objective is to investigate global measures of prosody and fluency as predictors of overall oral proficiency, fluency, and ...
Kurimo, Mikko; Getman, Yaroslav; Voskoboinik, Ekaterina; Al-Ghezi, Ragheb; Kallio, Heini; Kuronen, Mikko; von Zansen, Anna; Hilden, Raili; Kronholm, Sirkku; Huhta, Ari; Linden, Krister (International Speech Communication Association, 2023)The development of large multilingual speech models provides the possibility to construct high-quality speech technology even for low-resource languages. In this paper, we present the speech data of L2 learners of Finnish ...
Differences in acoustically determined sentence stress between native and L2 speakers of Finland Swedish Kallio, Heini; Kuronen, Mikko; Kautonen, Maria (Lund University, 2021)This study describes a pilot attempt to use acoustically determined sentence stress in distinguishing native and L2 speakers of Finland Swedish at different proficiency levels. The study is part of the DigiTala project ...
Towards the phonetic basis of spoken second language assessment : temporal features as indicators of perceived proficiency level Kallio, Heini; Šimko, Juraj; Huhta, Ari; Karhila, Reima; Vainio, Martti; Lindroos, Erik; Hildén, Raili; Kurimo, Mikko (Suomen soveltavan kielitieteen yhdistys, 2017)
Kautonen, Maria; von Zansen, Anna (Soveltavan kielentutkimuksen keskus, Jyväskylän yliopisto; Kielikoulutuspolitiikan verkosto, 2020)In this article, we discuss the premise of assessing oral language skills with the help of automatic speech recognition. We present the main goals and working methods in DigiTala project (2019–2023) – a research project ...