Beyond literacy and competency : The effects of raters’ perceived uncertainty on assessment of writing
Honko, M., Neittaanmäki, R., Jarvis, S., & Huhta, A. (2023). Beyond literacy and competency : The effects of raters’ perceived uncertainty on assessment of writing. Assessing Writing, 57, Article 100768. https://doi.org/10.1016/j.asw.2023.100768
Julkaistu sarjassa
Assessing WritingPäivämäärä
2023Oppiaine
LaskentatoimiSoveltava kielentutkimusHyvinvoinnin tutkimuksen yhteisöMonilukutaito oppimisessa ja yhteiskunnallisessa osallisuudessaYleiset kielitutkinnnotAccountingApplied language studiesSchool of WellbeingMultiliteracies for social participation and in learning across the life spanNational Certificates of Language ProficiencyTekijänoikeudet
© 2023 the Authors
This study investigated how common raters’ experiences of uncertainty in high-stakes testing are before, during, and after the rating of writing performances, what these feelings of uncertainty are, and what reasons might underlie such feelings. We also examined if uncertainty was related to raters’ rating experience or to the quality of their ratings. The data were gathered from the writing raters (n = 23) in the Finnish National Certificates of Proficiency, a standardized Finnish high-stakes language examination. The data comprise 12,118 ratings as well as raters’ survey responses and notes during rating sessions. The responses were analyzed by using thematic content analysis and the ratings by descriptive statistics and Many-Facets Rasch analyses. The results show that uncertainty is variable and individual, and that even highly experienced raters can feel unsure about (some of) their ratings. However, uncertainty was not related to rating quality (consistency or severity/leniency). Nor did uncertainty diminish with growing experience. Uncertainty during actual ratings was typically associated with the characteristics of the rated performances but also with other, more general and rater-related or situational factors. Other reasons external to the rating session were also identified for uncertainty, such as those related to the raters themselves. An analysis of the double-rated performances shows that although similar performance-related reasons seemed to cause uncertainty for different raters, their uncertainty was largely associated with different test-takers’ performances. While uncertainty can be seen as a natural part of holistic ratings in high-stakes tests, the study shows that even if uncertainty is not associated with the quality of ratings, we should constantly seek ways to address uncertainty in language testing, for example by developing rating scales and rater training. This may make raters’ work easier and less burdensome.
...
Julkaisija
Elsevier BVISSN Hae Julkaisufoorumista
1075-2935Asiasanat
Julkaisu tutkimustietojärjestelmässä
https://converis.jyu.fi/converis/portal/detail/Publication/184082105
Metadata
Näytä kaikki kuvailutiedotKokoelmat
Lisenssi
Samankaltainen aineisto
Näytetään aineistoja, joilla on samankaltainen nimeke tai asiasanat.
-
Arvioijakäyttäytymisen mittari: tutkimuskyselyt sekä muistiinpanotaulukko Arvioinnin epävarmuus ja epävarmuuden vaikutukset arvioinnin laatuun Yleisissä kielitutkinnoissa (YKI) -tutkimuksesta
Honko, Mari; Neittaanmäki, Reeta; Huhta, Ari; Jarvis, Scott (University of Jyväskylä, 2023)This research instrument was developed for a study on the uncertainty experienced by language raters and the impact of uncertainty on the quality of assessment in the official language examination, the National Certificates ... -
Arvioinnin epävarmuus ja epävarmuuden vaikutukset arvioinnin laatuun Yleisissä kielitutkinnoissa (YKI): tutkimusaineisto
Honko, Mari; Neittaanmäki, Reeta; Huhta, Ari; Jarvis, Scott (University of Jyväskylä, 2023)Tutkimuksessa tarkastellaan kielitaidon arvioijien kokemaa epävarmuutta sekä epävarmuuden vaikutusta arvioinnin laatuun Yleisissä kielitutkinnoissa (YKI). Aineisto kerättiin vuosina 2021-2022. Tutkimukseen osallistui ... -
Arvioinnin epävarmuus ja sen vaikutukset arvioinnin laatuun suomen kielen puhumisen taidon testiarvioinnissa
Honko, Mari; Neittaanmäki, Reeta (Puheen ja kielen tutkimuksen yhdistys ry, 2023)Tutkimuksessa tarkastellaan koulutettujen kielitaidon arvioijien arviointivarmuutta ja sen vaikutusta arvioinnin laatuun aikuisten suomenoppijoiden puhumisen taidon testiarvioinnissa. Aineisto koostuu 12 059 tehtäväkohtaisesta ... -
Second and foreign language fluency from a cognitive perspective : inefficiency and control of attention in lexical access
Olkkonen, Sanna (University of Jyväskylä, 2017)The present dissertation focuses on the fluency of second and foreign language (L2). Second and foreign language use is often assessed by its fluency. However, the connections between language proficiency and fluency ... -
DigiTala research project : automatic speech recognition in assessing L2 speaking
Kautonen, Maria; von Zansen, Anna (Soveltavan kielentutkimuksen keskus, Jyväskylän yliopisto; Kielikoulutuspolitiikan verkosto, 2020)In this article, we discuss the premise of assessing oral language skills with the help of automatic speech recognition. We present the main goals and working methods in DigiTala project (2019–2023) – a research project ...
Ellei toisin mainittu, julkisesti saatavilla olevia JYX-metatietoja (poislukien tiivistelmät) saa vapaasti uudelleenkäyttää CC0-lisenssillä.