Beyond literacy and competency : The effects of raters’ perceived uncertainty on assessment of writing
Honko, M., Neittaanmäki, R., Jarvis, S., & Huhta, A. (2023). Beyond literacy and competency : The effects of raters’ perceived uncertainty on assessment of writing. Assessing Writing, 57, Article 100768. https://doi.org/10.1016/j.asw.2023.100768
Published in
Assessing WritingDate
2023Discipline
LaskentatoimiSoveltava kielentutkimusHyvinvoinnin tutkimuksen yhteisöMonilukutaito oppimisessa ja yhteiskunnallisessa osallisuudessaYleiset kielitutkinnnotAccountingApplied language studiesSchool of WellbeingMultiliteracies for social participation and in learning across the life spanNational Certificates of Language ProficiencyCopyright
© 2023 the Authors
This study investigated how common raters’ experiences of uncertainty in high-stakes testing are before, during, and after the rating of writing performances, what these feelings of uncertainty are, and what reasons might underlie such feelings. We also examined if uncertainty was related to raters’ rating experience or to the quality of their ratings. The data were gathered from the writing raters (n = 23) in the Finnish National Certificates of Proficiency, a standardized Finnish high-stakes language examination. The data comprise 12,118 ratings as well as raters’ survey responses and notes during rating sessions. The responses were analyzed by using thematic content analysis and the ratings by descriptive statistics and Many-Facets Rasch analyses. The results show that uncertainty is variable and individual, and that even highly experienced raters can feel unsure about (some of) their ratings. However, uncertainty was not related to rating quality (consistency or severity/leniency). Nor did uncertainty diminish with growing experience. Uncertainty during actual ratings was typically associated with the characteristics of the rated performances but also with other, more general and rater-related or situational factors. Other reasons external to the rating session were also identified for uncertainty, such as those related to the raters themselves. An analysis of the double-rated performances shows that although similar performance-related reasons seemed to cause uncertainty for different raters, their uncertainty was largely associated with different test-takers’ performances. While uncertainty can be seen as a natural part of holistic ratings in high-stakes tests, the study shows that even if uncertainty is not associated with the quality of ratings, we should constantly seek ways to address uncertainty in language testing, for example by developing rating scales and rater training. This may make raters’ work easier and less burdensome.
...


Publisher
Elsevier BVISSN Search the Publication Forum
1075-2935Keywords
Publication in research information system
https://converis.jyu.fi/converis/portal/detail/Publication/184082105
Metadata
Show full item recordCollections
License
Related items
Showing items with similar title or keywords.
-
Instrument to study rater behaviour: Rater uncertainty and its impact on the quality of assessment in the official language examination, the National Certificates of Language Proficiency (YKI) project
Honko, Mari; Neittaanmäki, Reeta; Huhta, Ari; Jarvis, Scott (University of Jyväskylä, 2023)This research instrument was developed for a study on the uncertainty experienced by language raters and the impact of uncertainty on the quality of assessment in the official language examination, the National Certificates ... -
Rater uncertainty and its impact on the quality of assessment in the official language examination, the National Certificates of Language Proficiency (YKI): research dataset
Honko, Mari; Neittaanmäki, Reeta; Huhta, Ari; Jarvis, Scott (University of Jyväskylä, 2023)Tutkimuksessa tarkastellaan kielitaidon arvioijien kokemaa epävarmuutta sekä epävarmuuden vaikutusta arvioinnin laatuun Yleisissä kielitutkinnoissa (YKI). Aineisto kerättiin vuosina 2021-2022. Tutkimukseen osallistui ... -
Second and foreign language fluency from a cognitive perspective : inefficiency and control of attention in lexical access
Olkkonen, Sanna (University of Jyväskylä, 2017)The present dissertation focuses on the fluency of second and foreign language (L2). Second and foreign language use is often assessed by its fluency. However, the connections between language proficiency and fluency ... -
Arvioinnin epävarmuus ja sen vaikutukset arvioinnin laatuun suomen kielen puhumisen taidon testiarvioinnissa
Honko, Mari; Neittaanmäki, Reeta (Puheen ja kielen tutkimuksen yhdistys ry, 2023)Tutkimuksessa tarkastellaan koulutettujen kielitaidon arvioijien arviointivarmuutta ja sen vaikutusta arvioinnin laatuun aikuisten suomenoppijoiden puhumisen taidon testiarvioinnissa. Aineisto koostuu 12 059 tehtäväkohtaisesta ... -
DigiTala research project : automatic speech recognition in assessing L2 speaking
Kautonen, Maria; von Zansen, Anna (Soveltavan kielentutkimuksen keskus, Jyväskylän yliopisto; Kielikoulutuspolitiikan verkosto, 2020)In this article, we discuss the premise of assessing oral language skills with the help of automatic speech recognition. We present the main goals and working methods in DigiTala project (2019–2023) – a research project ...