Puheen ymmärtämisen tehtävien taitotasoille asettaminen Yleisissä kielitutkinnoissa

The aim of this study is to explore rater effects, such as consistency and severity of the judgements of 12 expert judges, and to report the extent of assessment criteria used by them. The data of the study are based on the three different phases of the standardization procedure which were analyzed using the MFRM model. In addition, feedback gathered from the expert judges was used to find out their opinions about the procedure and to validate the results of the standardization. The results of this study indicate that general discussion on and statistical information about the items influenced judges’ consistency and severity and their adaptation of the assessment criteria. The judges themselves also found that these factors affected their judgements. In addition, the nature of listening and the criteria caused difficulties during the procedure.