Adapting the Person Fit Analysis: Ideas on Detecting Person Misfit in Computerized Adaptive Testing

Abstract

In this editorial chapter, I aim to summarize findings on person fit analysis in computerized adaptive testing (CAT) from prior research and discuss potential avenues for further research. In item response theory (IRT) applications, person fit quantifies fit of a response pattern to the model (Bradlow & Weiss, 2001, p. 86). Person misfit refers to unexpected response patterns by individuals. There are many potential reasons of misfit including special knowledge (Sinharay, 2016), cheating, guessing (Meijer, 1996), fatigue (Swearingen, 1998), warming up (Meijer, 1996), or faking (Ferrando & Anguiano-Carrasco, 2012). Evaluation of misfit is a significant step for addressing discrepancies within the measurement model. When IRT models are used, evidence of model fit which involves person fit analysis results should be reported (Standard 4.10; AERA, APA & NCME, 2014) as validity evidence to enhance score interpretations. Once misfitting items are identified, corrective steps such as item revision or removal can be implemented. For examinees who exhibit misfit, additional exploration can be undertaken to pinpoint behaviors that might necessitate adjustments to the test program or corrective interventions for particular examinees.

Keywords

• Person Fit
• Computerized adaptive testing
• CUSUM approach

References

1. American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Bradlow, E. T., & Weiss, R. E. (2001). Outlier measures and norming methods for computerized adaptive tests. Journal of Educational and Behavioral Statistics, 26(1), 85-104. https://doi.org/10.3102/10769986026001085
2. Chen, J., & Gupta, A. K. (2012). Parametric statistical change point analysis: With applications to genetics, medicine, and finance (2nd ed.). Springer.
3. Csörgő, M., Horváth, L., & Szyszkowicz, B. (1997). Integral tests for suprema of Kiefer processes with application. Statistics & Risk Modeling, 15(4), 365-378. https://doi.org/10.1524/strm.1997.15.4.365
4. Ferrando, P. J., & Anguiano-Carrasco, C. (2012). Response Certainly, Conscienciousness, and Self-concept Clarity as antecedents of Acquiescence: A prediction model. Anuario de Psicología, 42(1), 103-112. https://psycnet.apa.org/record/2014-14293-007
5. Hendrawan, I., Glas, C. A., & Meijer, R. R. (2005). The effect of person misfit on classification decisions. Applied Psychological Measurement, 29(1), 26-44. https://doi.org/10.3390/educsci10110324
6. Maurelli, V. A., & Weiss, D. J. (1981). Factors Influencing the Psychometric Characteristics of an Adaptive Testing Strategy for Test Batteries. http://files.eric.ed.gov/fulltext/ED212676.pdf
7. McLeod, L. D., & Lewis, C. (1999). Detecting item memorization in the CAT environment. Applied Psychological Measurement, 23(2), 147-160. https://doi.org/10.1177/01466219922031275
8. ,Meijer, R. R. (1996). Person-fit research: An introduction. Applied Measurement in Education, 9(1), 3-8. https://psycnet.apa.org/doi/10.1207/s15324818ame0901_2

9. Meijer, R.R., van Krimpen-Stoop, E.M.L.A. (2009). Detecting Person Misfit in Adaptive Testing. In: van der Linden, W., Glas, C. (eds) Elements of Adaptive Testing. Statistics for Social and Behavioral Sciences. Springer, New York, NY. https://doi.org/10.1007/978-0-387-85461-8_16
10. Robin, F. (2002). Investigating the relationship between test response behavior, measurement and person fit. In annual meeting of the National Council on Measurement in Education, New Orleans, LA.
11. Sideridis, G., Ghamdi, H., & Zamil, O. (2023). Contrasting multistage and computer-based testing: score accuracy and aberrant responding. Frontiers in Psychology, 14, 1288177. https://doi.org/10.3389/fpsyg.2023.1288177
12. Sinharay, S. (2016). Person fit analysis in computerized adaptive testing using tests for a change point. Journal of Educational and Behavioral Statistics, 41(5), 521-549. https://doi.org/10.3102/1076998616658331
13. Smith, R. M., & Plackner, C. (2010). The family approach to assessing fit in Rasch measurement. In M. Garner, G. Engelhard Jr., W. Fisher Jr., & M. Wilson (Eds.), Advances in Rasch measurement (Vol. 1, pp. 64–85). JAM Press.
14. Trabin, T. E., & Weiss, D. J. (1979). The person response curve: Fit of individuals to item characteristic curve models (Research Report 79-7). Minneapolis: University of Minnesota, Department of Psychology, Psychometric Methods Program. https://apps.dtic.mil/sti/tr/pdf/ADA080933.pdf
15. van Krimpen-Stoop, E.M.L.A., & Meijer, R. R. (2000). Detecting person misfit in adaptive testing using statistical process control techniques J. van der Linden & C.A.W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 201-219). Kluwer-Nijhoff.
16. van Krimpen-Stoop, E.M.L.A., & Meijer, R. R. (1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327-345. https://psycnet.apa.org/doi/10.1177/01466219922031446
17. Walker, A. A., & Engelhard, G. (2016). Using person fit and person response functions to examine the validity of person scores in computer adaptive tests. In Pacific Rim Objective Measurement Symposium (PROMS) 2015 Conference Proceedings (pp. 369-381). Springer Singapore. http://dx.doi.org/10.1007/978-981-10-1687-5