Analyzing the Interaction of Item Position Effect and Student Characteristics within Explanatory IRT Models

Year 2022, Volume: 13 Issue: 4, 282 - 304, 25.12.2022

Sinem Demirkol , Hülya Kelecioğlu

https://doi.org/10.21031/epod.1126368

Cited By: 1

Abstract

The location of the same item in different positions among booklets leads to a biased estimation of item parameters. This undesirable effect on the probability of answering the items correctly is referred as the item position effect. The purpose of this study is to examine the items that are more sensitive to the item position effect and to investigate the student characteristics related to the item position effect. In the study, the items in the PISA 2015 reading domain are used. The study group consists of 2418 students who responded to the items in the reading domain from PISA 2015 Turkey Sample. Explanatory IRT models are used in the analysis of the research. According to the results, 42% of the items are affected by the item position. The most important characteristic related to item position is the SES level of students. In addition, male students are more affected by item position than female students.

Keywords

Item position effect, Explanatory IRT models, student characteristics reading domain

References

• Albano, A. D. (2013). Multilevel modeling of item position effects. Journal of Educational Measurement, 50(4), 408–426. https://doi.org/10.1111/jedm.12026
• Alexandrowicz, R., & Matschinger, H. (2008). Estimating item location effects by means of a generalized logistic regression model. Psychology Science Quarterly, 50(1), 64-74. https://www.psychologie-aktuell.com/fileadmin/download/PschologyScience/1-2008/06_Alexandrowicz.pdf
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME]. (2014). Standards for educational and psychological testing. American Psychological Association.
• Angoff, W. H. (1971). Scales, norms and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508-600). American Council on Education.
• Atar, B. (2011). An application of descriptive and explanatory item response models to TIMSS 2007 Turkey mathematics data. Education and Science, 36(159), 256-259. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/811
• Atar, B., & Cobanoglu Aktan, D. (2013). Person explanatory item response theory analysis: Latent regression two parameter logistic model. Education and Science, 38(168), 59–68. http://egitimvebilim.ted.org.tr/index.php/EB/article/view/942
• Bates, D., Maechler, M., Bokler, B., & Walker, S. (2014). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
• Berger, V. F., Munz, D. C., Smouse, A. D., & Angelino, H. (1969). The effects of item difficulty sequencing and anxiety reaction type on aptitude test performance. Journal of Psychology, 71(2), 253–258. https://doi.org/10.1080/00223980.1969.10543091
• Brennan, R. L. (1992). The context of context effects. Applied Measurement in Education, 5(3), 225–264. https://doi.org/10.1207/s15324818ame0503_4
• Briggs, D. C. (2008). Using explanatory item response models to analyze group differences in science achievement. Applied Measurement in Education, 21(2), 89-118. https://doi.org/10.1080/08957340801926086
• Bulut, O. (2021). eirm: Explanatory item response modeling for dichotomous and polytomous item responses, R package (version 0.3.0) [Computer software]. http://CRAN.R-project.org/package=eirm
• Bulut, O., Quo, O., & Gierl, M. (2017). A structural equation modeling approach for examining position effects in large‑scale assessments. Large Scale in Assessments in Education, 5(8), 1-20. https://doi.org/10.1186/s40536-017-0042-x
• Butler, J., & Adams, R. J. (2007). The impact of differential investment of student effort on the outcomes of international studies. Journal of Applied Measurement, 8(3), 279–304. https://pubmed.ncbi.nlm.nih.gov/17804895/
• Christiansen, A., & Janssen, R. (2020). Item position effects in listening but not in reading in the European Survey of Language Competences. Educational Assessment, Evaluation and Accountability, 33(3), 49–69. https://doi.org/10.1007/s11092-020-09335-7
• Cronbach, L. J. (1984). Essentials of psychological testing. Harper and Row.
• Davey, T., & Lee, Y. H. (2011). Potential impact of context effects on the scoring and equating of the multistage GRE revised general test (RR-11-26). ETS.
• De Boeck, P., & Wilson, M. (2004). Explanatory item response models: A generalized linear and nonlinear approach. Statistics for Social Science and Public Policy. Springer.
• Debeer, D., & Janssen, R. (2013). Modeling item-position effects within an IRT framework. Journal of Educational Measurement, 50(2), 164-185. https://doi.org/10.1111/jedm.12009
• DeMars, C. (2016). Madde tepki kuramı (Çev. Ed. H. Kelecioğlu). Nobel.
• Demirkol, S., & Kelecioğlu, H. (2022). Investigating the effect of item position on person and item parameters: PISA 2015 Turkey sample. Journal of Measurement and Evaluation in Education and Psychology, 13(1), 69-85. https://doi.org/10.21031/epod.958576
• Duncan, G. J., & Magnuson, K. A. (2005). Can family socioeconomic resources account for racial and ethnic test score gaps? The Future of Children, 15(1), 35-54. https://doi.org/10.1353/foc.2005.0004
• Eklöf, H. (2007). Test-taking motivation and mathematics performance in TIMSS 2003. International Journal of Testing, 7(3), 311–326. https://doi.org/10.1080/15305050701438074
• Embretson, S. E., & Reise, S. P. (2000). Item response theory (1st ed.). Psychology Press. https://doi.org/10.4324/9781410605269
• Finn, B. (2015). Measuring motivation in low-stakes assessments (RR-15–19). Educational Testing Service. https://doi.org/10.1002/ets2.12067
• Grandy, J. (1987). Characteristics of examinees who leave questions unanswered on the GRE general test under rights-only scoring (RR-87-38). Educational Testing Service. https://doi.org/10.1002/j.2330-8516.1987.tb00242.x
• Guertin, W. H. (1954). The effect of instructions and item order on the arithmetic subtest of the Wechsler- Bellevue. Journal of Genetic Psychology, 85(1), 79–83. https://doi.org/10.1080/00221325.1954.10532863
• Hahne, J. (2008). Analyzing position effects within reasoning items using the LLTM for structurally incomplete data. Psychology Science Quarterly, 50(3), 379-390. http://www.fyhe.com.au/past_papers/2006/Papers/Taylor.pdf
• Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Theory and applications. Kluwer-Nijhoff.
• Hartig, J., & Buchholz, J. (2012). A multilevel item response model for item position effects and individual persistence. Psychological Test and Assessment Modeling, 54(4), 418-431. https://www.proquest.com/openview/075b5103d499407933e3c62cca521618/1?pq-origsite=gscholar&cbl=43472
• Hohensinn, C., Kubinger, K., Reif, M., Schleicher, E., & Khorramdel, L. (2011). Analysing item position effects due to test booklet design within large-scale assessment. Educational Research and Evaluation, 17(6), 497-509. https://doi.org/10.1080/13803611.2011.632668
• Kingston, N. M., & Dorans, N. J. (1982). The effect of the position of an item within a test on item responding behavior: An analysis based on item response theory (RR-79-12bP). Educational Testing Service. https://doi.org/10.1002/j.2333-8504.1982.tb01308.x
• Kingston, N. M., & Dorans, N. J. (1984). Item location effects and their implications for IRT equating and adaptive testing. Applied Psychological Measurement, 8(2), 147–154. https://doi.org/10.1177/014662168400800202
• Leary, L. F., & Dorans, N. J. (1985). Implications for altering the context in which test items appear: A historical perspective on an immediate concern. Review of Educational Research, 55(3), 387–413. https://www.jstor.org/stable/1170392
• MacNicol, K. (1956). Effects of varying order of item difficulty in an unspeeded verbal test. Unpublished manuscript, Educational Testing Service.
• Meyers, J. L., Miller, G. E., & Way, W. D. (2009). Item position and item difficulty change in an IRT-based common item equating design. Applied Measurement in Education, 22(1), 38-60. https://doi.org/10.1080/08957340802558342
• McKeachie, W. J., Pollie, D., & Speisman, J. (1955). Relieving anxiety in classroom examinations. Journal of Abnormal and Social Psychology, 50(1), 93-98. https://doi.org/10.1037/h0046560
• Mollenkopf, W. G. (1950). An experimental study of the effects on item-analysis data of changing item placement and test time limit. Psychometrika, 15(3), 291–315. https://doi.org/10.1007/BF02289044
• Nagy, G., Nagengast, B., Frey, A., Becker, M., & Rose, N. (2018). A multilevel study of position effects in PISA achievement tests: Student- and school-level predictors in the German tracked school system. Assessment in Education: Principles, Policy & Practice, 26(4), 422–443. https://doi.org/10.1080/0969594X.2018.1449100
• Organisation for Economic Co-operation and Development [OECD]. (2017). PISA 2015 technical report. OECD. https://www.oecd.org/pisa/data/2015-technical-report/
• Pomplun, M., & Ritchie, T. (2004). An investigation of context effects for item randomization within testlets. Journal of Educational Computing Research, 30(3), 243–254. https://doi.org/10.2190/Y4FU-45V7-74UN-HW4T
• Qian, J. (2014). An investigation of position effects in large-scale writing assessments. Applied Psychological Measurement, 38(7), 518–534. https://doi.org/10.1177/0146621614534312
• Rose, N., Nagy, G., Nagengast, B., Frey, A., & Becker, M. (2019). Modeling multiple item context effects with generalized linear mixed models. Frontiers in Psychology, 10(248), 1-13. https://doi.org/10.3389/fpsyg.2019.00248
• Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of Educational Research, 75(3), 417–453. https://doi.org/10.3102/00346543075003417
• Smouse, A. D., & Munz, D. C. (1968). The effects of anxiety and item difficulty sequence on achievement testing scores. Journal of Psychology, 68(2), 181-184. https://doi.org/10.1080/00223980.1968.10543421 Stanley, J. C. (1961). Studying status vs. manipulating variables. In R. O. Collier & S. M. Elam (Eds.), Research design and analysis. Phi Delta Kappan.
• Taylor, J. A. (2005). Poverty and student achievement. Multicultural Education, 12(4), 53-55. https://www.proquest.com/openview/3c9d66c77504e21370c0b718cf66e27f/1?pq-origsite=gscholar&cbl=33246
• Towle, N. J., & Merrill, P. F. (1975). Effects of anxiety type and item-difficulty sequencing on mathematics test performance. Journal of Educational Measurement, 12(4), 241–249. https://www.jstor.org/stable/1434151 Weirich, S., Hecht, M., Penk, C., Roppelt, A., & Böhme, K. (2017). Item position effects are moderated by changes in test-taking effort. Applied Psychological Measurement, 41(2), 115–129. https://doi.org/10.1177/0146621616676791
• Whitely, E., & Dawis, R. (1976). The influence of test context on item difficulty. Educational and Psychological Measurement, 36(2), 329–337. https://doi.org/10.1177/001316447603600211
• Wise, L. L., Chia, W. J., & Park, R. (1989, March 27-31). Item position effects for test of word knowledge and arithmetic reasoning. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
• Wise, S. L., & DeMars, C. E. (2005). Examinee motivation in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://doi.org/10.1207/s15326977ea1001_1
• Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163-183. https://doi.org/10.1207/s15324818ame1802_2
• Wu, Q., Debeer, D. Buchholz, J., Hartig, J., & Janssen, R. (2019). Predictors of individual performance changes related to item positions in PISA assessments. Large Scale Assessment in Education, 7(5), 1-20. https://doi.org/10.1186/s40536-019-0073-6
• Yen, W. M. (1980). The extent, causes and importance of context effects on item parameters for two latent trait models. Journal of Educational Measurement, 17(4), 297–311. http://www.jstor.org/stable/1434871