The Effect of Test-taking Familiarity on Standard Error of the Measurement in Computerized Adaptive Tests

Abstract

Computerized Adaptive Tests (CAT) tailor item difficulty to the examinee's ability, potentially reducing the standard error of measurement (SEM) even while administering fewer items. As one of the main goals of an assessment system is to minimize SEM and to provide ability estimations closest to the true score of examinees, investigating factors related to SEM provides valuable information. As CAT is a comparably new testing system in classroom use, test-taking familiarity might be related to SEM in CAT, especially for younger students. However, most of the CAT studies related to SEM use simulated data and experimental studies on the live application of CAT in classroom setting are missing. Thus, the current study investigates the impact of test-taking familiarity on SEM in a CAT consisting of 4th-grade mathematics items. This experimental study involved 403 students assigned randomly into control and experimental groups. The experimental group received easier warm-up items before the CAT while the control group began the CAT without warm-up items. Results indicated that test-taking familiarity significantly reduced SEM (control group M=0.369; experimental group M=0.342; η2 = .579). The results revealed that students who began with warm-up items had lower SEM. Thus, the current study suggests that using warm-up items in CAT is likely to increase the precision of ability estimations especially for younger students.

Keywords

• Live Computerized Adaptive Tests
• Test-taking Familiarity
• Standard Error of the Measurement
• Experimental Study

References

1. Anastasi, A. (1981). Coaching, test sophistication, and developed abilities. American Psychologist, 36(10), 1086–1093. https://doi.org/10.1037/0003-066X.36.10.1086
2. Chalmers, R. P. (2012). mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software, 48, 1–29. https://doi.org/10.18637/jss.v048.i06
3. Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd edition). Routledge.
4. Drasgow, F., Luecht, R. M., & Bennet, R. (2006). Technology and Testing. In R. L. Brennan (Ed.), Educational Measurement (4th ed., pp. 471–515). American Council on Education/Praeger Publishers
5. Embretson, S. E. (1996). The new rules of measurement. Psychological Assessment, 8(4), 341-349. https://doi.org/10.1037/1040-3590.8.4.341
6. Field, A. (2018). Discovering Statistics Using IBM SPSS Statistics. Sage Publication.
7. Frey, A., Liu, T., Fink, A., & König, C. (2024). Meta-analysis of the effects of computerized adaptive testing on the motivation and emotion of examinees. European Journal of Psychological Assessment, 40(4), 427–443. https://doi.org/10.1027/1015-5759/a000821
8. Fritts, B. E., & Marszalek, J. M. (2010). Computerized Adaptive Testing, Anxiety Levels, and Gender Differences. Social Psychology of Education: An International Journal, 13(3), 441–458. https://doi.org/10.1007/s11218-010-9113-3

9. Gershon, R. C. (1992). Test Anxiety and Item Order: New Concerns for Item Response Theory. In M Wilson (Ed.) Objective Measurement: Theory into Practice. Vol. 1. Ablex, Norwood NJ.
10. Glas, C. A. W., & Geerlings, H. (2009). Psychometric aspects of pupil monitoring systems. Studies in Educational Evaluation, 35, 83-88. https://doi.org/10.1016/j.stueduc.2009.10.006
11. Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational measurement: issues and practice, 12(3), 38-47.
12. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory (Vol. 2). Sage
13. Huang, H. T. D., Hung, S. T. A., Chao, H. Y., Chen, J. H., Lin, T. P., & Shih, C. L. (2022). Developing and validating a computerized adaptive testing system for measuring the English proficiency of Taiwanese EFL university students. Language Assessment Quarterly, 19(2), 162–188. https://doi.org/10.1080/15434303.2021.1984490
14. Istiyono, E., Dwandaru, W. S. B., Setiawan, R., & Megawati, I. (2020). Developing of Computerized Adaptive Testing to Measure Physics Higher Order Thinking Skills of Senior High School Students and Its Feasibility of Use. European Journal of Educational Research, 9(1), 91-101. https://doi.org/10.12973/eu-jer.9.1.91
15. Kulik, J. A., Kulik, C.-L. C., & Bangert, R. L. (1984). Effects of Practice on Aptitude and Achievement Test Scores. American Educational Research Journal, 21(2), 435–447. https://doi.org/10.2307/1162453
16. Lord, F. M. (1962). Estimating norms by item-sampling. Educational and Psychological Measurement, 22(2), 259–267. https://doi.org/10.1177/001316446202200202
17. Lord, F. M. (1980). Applications of Item Response Theory to Practical Testing Problems. Routledge.
18. Maeda, H. (2019). The Effects of Test Familiarity on Person-Fit and Aberrant Behavior. Proceedings of the 2019 AERA Annual Meeting. 2019 AERA Annual Meeting. https://doi.org/10.3102/1508730
19. Martin, A. J., & Lazendic, G. (2018). Computer-adaptive testing: Implications for students’ achievement, motivation, engagement, and subjective test experience. Journal of educational psychology, 110(1), 27-45. https://doi.org/10.1037/edu0000205
20. Matteucci, M., & Veldkamp, B. P. (2013). On the use of MCMC computerized adaptive testing with empirical prior information to improve efficiency. Statistical Methods & Applications, 22(2), 243–267. https://doi.10.1007/s10260-012-0216-1
21. Mizumoto, A., Sasao, Y., & Webb, S. A. (2019). Developing and evaluating a computerized adaptive testing version of the Word Part Levels Test. Language Testing, 36(1), 101–123. https://doi.org/10.1177/0265532217725776
22. Moreno, K. E. (1997). CAT-ASVAB operational test and evaluation. In W. A. Sands, B. K. Waters, & R. McBride (Eds.), Computerized adaptive testing: From inquiry to operation (pp. 199–205). Washington, DC: American Psychological Association
23. Mullis, I. V., Martin, M. O., & von Davier, M. (2021). TIMSS 2023 Assessment Frameworks. International Association for the Evaluation of Educational Achievement. Chestnut Hill, MA: TIMSS & PIRLS International Study Center and International Association for the Evaluation of Educational Achievement (IEA).
24. Olson, L. (2003). Legal twists, digital turns: Computerized testing feels the impact of “No Child Left Behind.” Education Week, 12(35), 11–14.
25. Reeve, C. L., Heggestad, E. D., & Lievens, F. (2009). Modeling the impact of test anxiety and test familiarity on the criterion-related validity of cognitive ability tests. Intelligence, 37(1), 34-41. https://doi.org/10.1016/j.intell.2008.05.003
26. Renaissance Learning. (2016). STAR Math technical manual. Renaissance Learning.
27. Scharfen, J., Peters, J. M., & Holling, H. (2018). Retest effects in cognitive ability tests: A meta-analysis. Intelligence, 67, 44-66. https://doi.org/10.1016/j.intell.2018.01.003
28. Shoemaker, D. M. (1973). Principles and Procedures of Multiple Matrix Sampling. Ballinger Publishing Company.
29. The Psychometric Centre. (2023). Concerto Platform version 5 [Software]. URL: https://concertoplatform.com
30. Thissen, D., & Mislevy, R. J. (2000). Testing Algorithms. In Howard Wainer, Computerized Adaptive Testing: A Primer (pp. 101–134). Lawrence Erlbaum Associates.
31. Thompson, N., & Weiss, D. (2011). A Framework for the Development of Computerized Adaptive Tests. Practical Assessment, Research, and Evaluation, 16(1), 1-9.
32. Walter, O. B., & Rose, M. (2013). Effect of item order on item calibration and item bank construction for computer adaptive tests. Psychological Test and Assessment Modeling, 55(1), 81-91
33. Weiss, D. J. (Ed.). (1980). Proceedings of the 1979 Computerized Adaptive Testing Conference. Minneapolis: University of Minnesota, Department of Psychology, Psychometric Methods Program, Computerized Adaptive Testing Laboratory.
34. Weiss, D. J. (1983). Latent trait theory and adaptive testing. In D. J. Weiss (Ed.). New horizons in testing (pp. 5-7). New York: Academic Press.
35. Weiss, D. J. (2004). Computerized Adaptive Testing for Effective and Efficient Measurement in Counseling and Education. Measurement and Evaluation in Counseling and Development, 37(2), 70–84. https://doi.org/10.1080/07481756.2004.11909751
36. Weiss, D. J. ve Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21, 361-375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x
37. van Der Linden, W. J. (1999). Empirical Initialization of the Trait Estimator in Adaptive Testing. Applied Psychological Measurement, 23(1), 21–29. https://doi:10.1177/01466219922031149
38. van der Linden, W. J. (2001). Computerized test construction. Research Report. Twente University, Enschede (Netherlands)
39. van der Linden, W. J., & Pashley, P. J. (2010). Item Selection and Ability Estimation in Adaptive Testing. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of Adaptive Testing (pp. 3–30). Springer New York. https://doi:10.1007/978-0-387-85461-8_1
40. Verschoor, A. J., & Straetmans, G. J. J. M. (2010). MATHCAT: A Flexible Testing System in Mathematics Education for Adults. In W. J. van der Linden & C. A. W. Glas (Eds.), Elements of Adaptive Testing (pp. 137–150). Springer Science & Business Media.
41. Zara, A. R. (1988). Introduction to item response theory and computerized adaptive testing as applied in licensure and certification testing. National Clearinghouse of Examination Information Newsletter, 6, 11–17.
42. Zeidner, M. (1998). Test anxiety: The state of the art. New York: Plenum.