The impact of rubric layout on L2 English writing scores and raters’ decision-making

Abstract

This study examined the influence of rubric layout on ratings assigned to L2 English essays and raters’ decision-making process in the context of Turkish higher education. Raters scored 24 essays in two rounds with the same analytic rubric presented in standard and reverse layouts, responded to a criteria importance survey at six time points throughout the study, and provided written score explanations to justify their ratings. The two scoring rounds were conducted eight weeks apart, with the rubric categories reversed in the second round. A two-way repeated measures ANOVA revealed significant differences in scores between the two rounds. Multivariate generalizability analyses indicated that the rubric layout influenced variability in scores across raters. Content analysis of the written score explanations revealed a primacy effect, with raters placing greater emphasis on organization and grammar when these categories appeared earlier. However, raters consistently prioritized content and organization when justifying their scores. These findings suggest that the placement of rubric categories can affect raters’ focus and scoring, emphasizing the importance of careful rubric design to ensure consistent assessment outcomes.

Keywords

• Primacy effect
• Analytic scoring
• Rater decision-making
• Rater variations

Project Number

2024-148

Ethical Statement

Ordu University, 2024-148.

References

1. Ahmadi, A., & Tavazoei, M. (2025). Rater training through eye-tracking: A case-study of a novice rater. Teaching English Language, 19(2), 511-539. https://doi.org/10.22132/tel.[phone]
2. Ballard, L. (2017). The effects of primacy on rater cognition: An eye-tracking study. [Unpublished doctoral dissertation]. Michigan State University. https://d.lib.msu.edu/etd/4632
3. Bacha, N. (2001). Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29(3), 371-383. https://doi.org/10.1016/S0346-251X(01)00025-2
4. Barkaoui, K. (2007). Rating scale impact on EFL essay marking: A mixed-method study. Assessing Writing, 12(2), 86-107. https://doi.org/10.1016/j.asw.2007.07.001
5. Barkaoui, K. (2010). Variability in ESL essay rating processes: The role of the rating scale and rater experience. Language Assessment Quarterly, 7(1), 54 74. https://doi.org/10.1080/15434300903464418
6. Brennan, R.L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
7. Crano, W.D. (1977). Primacy versus recency in retention of information and opinion change. The Journal of Social Psychology, 101(1), 87-96. https://doi.org/10.1080/00224545.1977.9923987
8. Cresswell, J.W. (2011). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). PHI Learning Private.

9. Cumming, A. (1990). Expertise in evaluating second language composition. Language Testing, 7(1), 31-51. https://doi.org/10.1177/02655322900070010
10. Cumming, A., Kantor, R., & Powers, D. (2002). Decision making while rating ESL/EFL writing tasks: A descriptive framework. Modern Language Journal, 86(1), 67-96. https://doi.org/10.1111/1540-4781.00137
11. DeRemer, M.L. (1998). Writing assessment: Raters’ elaboration of the rating task. Assessing Writing, 5(1), 7-29. https://doi.org/10.1016/S1075-2935(99)80003-8
12. Diab, R., & Balaa, L. (2012). Developing detailed rubrics for assessing critique writing: Impact on EFL university students’ performance and attitudes. TESOL Journal, 2(1), 52 72. https://doi.org/10.5054/tj.2011.244132
13. Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155-185. https://doi.org/10.1177/0265532207086780
14. Eckes, T. (2012). Operational rater types in writing assessment: Linking rater cognition to rater behavior. Language Assessment Quarterly, 9(3), 270-292. https://doi.org/10.1080/15434303.2011.649381
15. Forgas, J.P. (2011). Can negative affect eliminate the power of first impressions? Affective influences on primacy and recency effects in impression formation. Journal of Experimental Social Psychology, 47(2), 425-429. http://doi.org/10.1016/j.jesp.2010.11.005
16. Gay, L.R., Mills, G.E., & Airasian, P. (2009). Educational research: Competencies for analysis and applications (9th ed.). Pearson Education Inc.
17. Goodwin, S. (2016). A Many-Facet Rasch analysis comparing essay rater behavior on an academic English reading/writing test used for two purposes. Assessing Writing, 30, 21 31. https://doi.org/10.1016/j.asw.2016.07.004
18. Goulden, N.R. (1994). Relationship of analytic and holistic methods to raters’ scores for speeches. The Journal of Research and Development in Education, 27(2), 73-82.
19. Heidari, N., Ghanbari, N., & Abbasi, A. (2022). Raters’ perceptions of rating scales criteria and its effect on the process and outcome of their rating. Language Testing in Asia, 12(1). https://doi.org/10.1186/s40468-022-00168-3
20. Huang, J. (2011). Generalizability theory as evidence of concerns about fairness in large‐scale ESL writing assessments. TESOL Journal, 2(4), 423-443. https://doi.org/10.5054/tj.2011.269751
21. Jeong, H. (2019). Writing scale effects on raters: An exploratory study. Language Testing in Asia, 9(20), 1-19. https://doi.org/10.1186/s40468-019-0097-4
22. Kim, H. (2020). Effects of rating criteria order on the halo effect in L2 writing assessment: A many-facet Rasch measurement analysis. Language Testing in Asia, 10(1), Article 16. https://doi.org/10.1186/s40468-020-00115-0
23. Lumley, T. (2005). Assessing second language writing: The rater’s perspective. Peter Lang.
24. Panadero, E., Delgado, P., Zamorano, D., Pinedo, L., Fernández-Ortube, A., & Barrenetxea-Mínguez, L. (2025). Putting excellence first: How rubric performance level order and feedback type influence students’ reading patterns and task performance. Learning and Instruction, 99, Article 102168. https://doi.org/10.1016/j.learninstruc.2025.102168
25. Perkins, K. (1983). On the use of composition scoring techniques, objective measures, and objective tests to evaluate ESL writing ability. TESOL Quarterly, 17(4), 651 671. https://doi.org/10.2307/3586618
26. Plakans, L. (2013). Writing scale development and use within a language program. TESOL Journal, 4(1), 151-163. https://doi.org/10.1002/tesj.66
27. Raimes, A. (1990). The TOEFL test of written English: Causes for concern. TESOL Quarterly, 24(3), 427-442. https://doi.org/10.2307/3587228
28. Rezaei, A.R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15(1), 18-39. https://doi.org/10.1016/j.asw.2010.01.003
29. Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A premier. Sage.
30. Song, B., & Caruso, I. (1996). Do English and ESL faculty differ in evaluating the essays of native English speaking, and ESL students? Journal of Second Language Writing, 5(2), 163 182. https://doi.org/10.1016/S1060-3743(96)90023-5
31. Şahan, Ö. (2019). The impact of rater experience and essay quality on the variability of EFL writing scores. In S. Papageorgiou & K.M. Bailey (Eds.), Global perspectives on language assessment: Research, theory, and practice (pp. 32-46). Routledge. https://doi.org/10.4324/9780429437922
32. Şahan, Ö., & Razı, S. (2020). Do experience and text quality matter for raters’ decision-making behaviors? Language Testing, 37(3), 311-332. https://doi.org/10.1177/0265532219900228
33. Tulving, E. (2008). On the law of primacy. In M.A. Gluck, J.R. Anderson, & S.M. Kosslyn (Eds.), Memory and mind: A festschrift for Gordon H. Bower (pp. 31-48). Lawrence Erlbaum Associates Publishers.
34. Vaughan, C. (1991). Holistic assessment: What goes on in the raters’ minds? In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 11-126). Ablex.
35. Weigle, S.C. (2002). Assessing writing. Cambridge University Press. https://doi.org/10.1017/CBO9780511732997
36. Winke, P., & Lim, H. (2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing, 25, 38-54. https://doi.org/10.1016/j.asw.2015.05.002
37. Wolfe, E.W., Kao, C., & Ranney, M. (1998). Cognitive differences in proficient and nonproficient essay scorers. Written Communication, 15(4), 465-492. https://doi.org/10.1177/0741088398015004002
38. Yan, T., Cernat, A., & Keusch, F. (2025). The impact of scale direction on data quality. Survey Research Methods, 19(2), 175-186. https://doi.org/10.18148/srm/2025.v19i2.8384