Beyond literacy and competency : The effects of raters’ perceived uncertainty on assessment of writing
Honko, M., Neittaanmäki, R., Jarvis, S., & Huhta, A. (2023). Beyond literacy and competency : The effects of raters’ perceived uncertainty on assessment of writing. Assessing Writing, 57, Article 100768. https://doi.org/10.1016/j.asw.2023.100768
Julkaistu sarjassa
Assessing WritingPäivämäärä
2023Oppiaine
LaskentatoimiSoveltava kielentutkimusHyvinvoinnin tutkimuksen yhteisöMonilukutaito oppimisessa ja yhteiskunnallisessa osallisuudessaYleiset kielitutkinnnotAccountingApplied language studiesSchool of WellbeingMultiliteracies for social participation and in learning across the life spanNational Certificates of Language ProficiencyTekijänoikeudet
© 2023 the Authors
This study investigated how common raters’ experiences of uncertainty in high-stakes testing are before, during, and after the rating of writing performances, what these feelings of uncertainty are, and what reasons might underlie such feelings. We also examined if uncertainty was related to raters’ rating experience or to the quality of their ratings. The data were gathered from the writing raters (n = 23) in the Finnish National Certificates of Proficiency, a standardized Finnish high-stakes language examination. The data comprise 12,118 ratings as well as raters’ survey responses and notes during rating sessions. The responses were analyzed by using thematic content analysis and the ratings by descriptive statistics and Many-Facets Rasch analyses. The results show that uncertainty is variable and individual, and that even highly experienced raters can feel unsure about (some of) their ratings. However, uncertainty was not related to rating quality (consistency or severity/leniency). Nor did uncertainty diminish with growing experience. Uncertainty during actual ratings was typically associated with the characteristics of the rated performances but also with other, more general and rater-related or situational factors. Other reasons external to the rating session were also identified for uncertainty, such as those related to the raters themselves. An analysis of the double-rated performances shows that although similar performance-related reasons seemed to cause uncertainty for different raters, their uncertainty was largely associated with different test-takers’ performances. While uncertainty can be seen as a natural part of holistic ratings in high-stakes tests, the study shows that even if uncertainty is not associated with the quality of ratings, we should constantly seek ways to address uncertainty in language testing, for example by developing rating scales and rater training. This may make raters’ work easier and less burdensome.
...
Julkaisija
Elsevier BVISSN Hae Julkaisufoorumista
1075-2935Asiasanat
Julkaisu tutkimustietojärjestelmässä
https://converis.jyu.fi/converis/portal/detail/Publication/184082105
Metadata
Näytä kaikki kuvailutiedotKokoelmat
Lisenssi
Samankaltainen aineisto
Näytetään aineistoja, joilla on samankaltainen nimeke tai asiasanat.
-
Instrument to study rater behaviour: Rater uncertainty and its impact on the quality of assessment in the official language examination, the National Certificates of Language Proficiency (YKI) project
Honko, Mari; Neittaanmäki, Reeta; Huhta, Ari; Jarvis, Scott (University of Jyväskylä, 2023)This research instrument was developed for a study on the uncertainty experienced by language raters and the impact of uncertainty on the quality of assessment in the official language examination, the National Certificates ... -
Rater uncertainty and its impact on the quality of assessment in the official language examination, the National Certificates of Language Proficiency (YKI): research dataset
Honko, Mari; Neittaanmäki, Reeta; Huhta, Ari; Jarvis, Scott (University of Jyväskylä, 2023)Tutkimuksessa tarkastellaan kielitaidon arvioijien kokemaa epävarmuutta sekä epävarmuuden vaikutusta arvioinnin laatuun Yleisissä kielitutkinnoissa (YKI). Aineisto kerättiin vuosina 2021-2022. Tutkimukseen osallistui ... -
Second and foreign language fluency from a cognitive perspective : inefficiency and control of attention in lexical access
Olkkonen, Sanna (University of Jyväskylä, 2017)The present dissertation focuses on the fluency of second and foreign language (L2). Second and foreign language use is often assessed by its fluency. However, the connections between language proficiency and fluency ... -
DigiTala research project : automatic speech recognition in assessing L2 speaking
Kautonen, Maria; von Zansen, Anna (Soveltavan kielentutkimuksen keskus, Jyväskylän yliopisto; Kielikoulutuspolitiikan verkosto, 2020)In this article, we discuss the premise of assessing oral language skills with the help of automatic speech recognition. We present the main goals and working methods in DigiTala project (2019–2023) – a research project ... -
Tilannetehtävät Yleisten kielitutkintojen suomen keskitason tutkinnon puhumisen osakokeessa
Ahola, Sari; Tossavainen, Henna (Suomen soveltavan kielitieteen yhdistys ry, 2013)One task type in the oral test in the Finnish National Certificates (NC) language testing system is reacting to situations. This paper reports a content analysis of the task rubrics of five situational tasks used in the ...
Ellei toisin mainittu, julkisesti saatavilla olevia JYX-metatietoja (poislukien tiivistelmät) saa vapaasti uudelleenkäyttää CC0-lisenssillä.