Comparison of Item Response Theory Scaling Methods with ROC Analysis

Abstract

In this study, one-dimensional item response theory models were evaluated using different scaling methods. In this context, the equating errors and the area under the curve of four scaling methods (Stocking-Lord, Heabara, Mean-Sigma, Mean-Mean), and one, two, and three parameters logistic models (1PL, 2PL, and 3PL) in non-equivalent groups with anchor test (NEAT) design were examined. Additionally, the equating errors of the scaling methods and the results obtained from ROC analysis were compared. Qatar's and Australia's PISA 2012 mathematical literacy test data were used in the study. The minimum error was obtained from the Mean-Mean method with the 1PL model, and the maximum error was obtained from the Mean-Mean method with the 3PL model. Similar results were observed in all comparisons and supported each other. It is concluded that ROC analysis can be used to compare different conditions, methods and models.

Keywords

• test equating
• ROC analysis
• scaling methods
• AUC

References

1. Aşiret, S., & Sünbül, S. Ö. (2016). Investigating test equating methods in small samples through various factors. Educational Sciences: Theory & Practice, 16(2), 647-668. https://doi.org/10.12738/estp.2016.2.2762
2. Baker, F. B., & Al-Karni, A. (1991). A comparison of two procedures for computing IRT equating coefficients. Journal of Educational Measurement, 28, 147-162. https://www.jstor.org/stable/1434796
3. Boduroğlu, E. (2017). The study of classification consistency of transition to higher education examination according to the cut-off scores obtained from different [Master’s Thesis, Mersin University]. https://tez.yok.gov.tr/UlusalTezMerkezi/
4. Branberg, K., & Wiberg, M. (2011). Observed score linear equating with covariates. Journal of Educational Measurement, 48(4), 419-440. https://www.jstor.org/stable/41427533
5. Carrington, A. M., Manuel, D. G., Fieguth, P. W., Ramsay, T., Osmani, V., Wernly, B., ... & Holzinger, A. (2021). Deep ROC analysis and AUC as balanced average accuracy to improve model selection, understanding and interpretation. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2022.3145392
6. Çokluk, Ö., Şekercioğlu, G., & Büyüköztürk, Ş. (2012). Sosyal bilimler için çok değişkenli istatistik: SPSS ve Lisrel uygulamaları. Pegem Akademi.
7. Embretson, S. (1996) The new rules of measurement. Psychological Assessment, 8(4), 341-349. https://doi.org/10.1037/1040-3590.8.4.341
8. Embretson, S. (1997). Multicomponent response models. In W. van der Linden & R. Hambleton (Eds.), Handbook of modern Item Response Theory (pp. 305-321). Springer-Verlag.

9. Embretson, S. E., & Reise, S. P. (2000). Item Response Theory for psychologists. Lawrence Elbaum Associates.
10. Faraggi, D., & Reiser, B. (2002). Estimation of the area under the ROC curve. Statistics in Medicine, 21, 3093-3106. https://doi.org/10.1002/sim.1228
11. Flach, P., Blockeel, H., Ferri, C., Hernandez-Orallo, J., & Struyf, J. (2003). Decision support for data mining: An introduction to ROC analysis and its applications. In D. Mladenić, N. Lavrač, M. Bohanec & S. Moyle (Eds.), Data mining and decision support: Integration and collaboration (vol. 745, pp. 81-90). Springer. https://doi.org/10.1007/978-1-4615-0286-9_7
12. Flach, P. (2019). Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward. Proceedings of the AAAI Conference on Artificial Intelligence, 33, 9808–9814. https://doi.org/10.1609/aaai.v33i01.33019808
13. Gao, X., Zhu, R., Chen, H., & Harris, D.J. (2008, March 25-27). Impact of anchor-item selections on IRT scale transformation and equating [Paper presentation]. Annual meeting of the National Council on Measurement in Education, New York.
14. Gialluca, K. A., Crichton, L. I., Vale, C. D., & Ree, M. J. (1984). Methods for equating mental tests (Report No. ED251512). ERIC. https://files.eric.ed.gov/fulltext/ED251512.pdf
15. Gonzalez, J. (2014). SNSequate: Standard and nonstandard statisticalmodels and methods for test equating. Journal of Statistical Software, 59(7), 1-30. https://doi.org/10.18637/jss.v059.i07
16. Haebara, T. (1980). Equating logistic ability scales by a weighted least squares method. Japanese Psychological Research, 22(3), 144-149. https://doi.org/10.4992/psycholres1954.22.144
17. Hajian-Tilaki, K. O., Hanley, J. A., Joseph, L., & Collet, J. P. (1997). A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Medical Decision Making, 17(1), 94-102. https://doi.org/10.1177/0272989X9701700111
18. Hajian-Tilaki, K. (2018). Receiver operator characteristic analysis of biomarkers evaluation in diagnostic research. Journal of Clinical and Diagnostic Research, 12(6), 1-8. https://doi.org/10.7860/JCDR/2018/32856.11609
19. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Kuluwer-Nijhoff Publisihing.
20. Hanley J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143, 29–36. https://doi.org/10.1148/radiology.143.1.7063747
21. Hanson, B. A., & Béguin, A. A. (2002). Obtaining a common scale for item response theory item parameters using separate versus concurrent estimation in the common-item equating design. Applied Psychological Measurement, 26(1), 3-24. https://doi.org/10.1177/0146621602026001001
22. Heagerty, P., Lumley, T., & Pepe, M. (2000). Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics, 56, 337-344. https://doi.org/10.1111/j.0006-341x.2000.00337.x
23. Hosmer, D. W., & Lemeshow, S. (2000). Applied logistic regression (2nd ed.). John Wiley & Sons
24. Jones, L., & Rushton, D. (2019, September 1-6). Optimising geotechnical correlations using receiver operating characteristic (ROC) analysis. The XVII European Conference on Soil Mechanics and Geotechnical Engineering (ECSMGE 2019), Reykjavik, Iceland.
25. Karaismailoğlu, E. (2015). The use of time dependent roc curve for evaluation of the performance of markers during follow-up time (Combined Doctoral Dissertation, Hacettepe University). https://tez.yok.gov.tr/UlusalTezMerkezi/
26. Kılıç, S. (2013). Klinik karar vermede ROC analizi. Journal of Mood Disorders, 3(3), 135-140. https://doi.org/10.5455/jmood.20130830051624
27. Kolen, M. J. (1988). An NCME instructional module on traditional equating methodology. Educational Measurement: Issues and Practice, 7, 29-36.
28. Kolen, M. J., & Brennan, R. L. (2014). Test equating, scaling, and linking: Methods and practices (3nd ed.). Springer.
29. Köksal, B. (2011). Model selection with ROC curve estimation in regression analysis [Master’s thesis, Marmara University]. https://tez.yok.gov.tr/UlusalTezMerkezi/
30. Lasko, T. A., Bhagwat, J. G., Zou, K. H., & Ohno-Machado, L. (2005). The use of receiver operating characteristic curves in biomedical informatics. Journal of Biomedical Informatics, 38(5), 404-415. https://doi.org/10.1016/j.jbi.2005.02.008
31. Liaw, A., & Wiener, M. (2018). randomForest: Breiman and Cutler's Random Forests for classification and regression. https://cran.r-project.org/web/packages/randomForest/randomForest.pdf
32. Livingston, S. A., & Lewis, C. (2009). Small-sample equating with prior information (Report No. RR-09-25). ETS. https://files.eric.ed.gov/fulltext/ED507811.pdf
33. Pardo, M. C., & Franco-Pereira, A.M. (2017). Non parametric ROC summary statistics. REVSTAT-Statistical Journal, 15(4), 583-600. https://eprints.ucm.es/id/eprint/46564/1/PardoCarmen29.pdf
34. Pepe, M., Janes, H., Longton, G., Leisenring, W., & Newcomb, P. (2004). Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. American Journal of Epidemiology, 159, 882-890. https://doi.org/10.1093/aje/kwh101
35. Pundir, S., & Amala, R. (2015). Evaluation of biomarker using two parameter bi-exponential ROC curve. Pakistan Journal of Statistics and Operation Research, 11(4), 481-496. https://doi.org/10.18187/pjsor.v11i4.992 Revelle, W. (2018). psych: Procedures for personality and psychological research. http://kambing.ui.ac.id/cran/web/packages/psych/psych.pdf
36. Rizopoulos, D. (2018). ltm: Latent Trait Models under IRT. https://cran.r-project.org/web/packages/ltm/ltm.pdf
37. Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisace k F, Sanchez J, Müller M (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 77. https://doi.org/10.1186/1471-2105-12-77
38. Senaratna, D. M., Sooriyarachchim, M. R., & Meyen, N. (2015). Bivariate test for testing the EQUALITY of the average areas under correlated receiver operating characteristic curves (Test for comparing of AUC’s of correlated ROC curves). American Journal of Applied Mathematics and Statistics, 3(5), 190-198. https://doi.org/10.12691/ajams-3-5-3
39. Swaving, M., van Houwelingen, H., Ottes, F. P., & Steerneman, T. (1996). Statistical comparison of ROC curves from multiple readers. Medical Decision Making, 16(2), 143-152. https://doi.org/10.1177/0272989X9601600206
40. Swets, J. A., Dawes, R. M., & Monahan, J. (2000). Better decisions through science. Scientific American, 283, 82-87. https://doi.org/10.1038/scientificamerican1000-82
41. Stocking, M. L., & Lord, F. M. (1983). Developing a common metric in item response theory. Applied Psychological Measurement, 7(2), 201-210. https://doi.org/10.1177 / 014662168300700208
42. Taşdemir, F., & Çokluk, Ö. ( 2013). Angoff (1-0), Nedelsky and examination of classification accuracies of a test by determination methods of limit values. Mediterranean Journal of Humanities, 3(2), 241-261. https://doi.org/10.13114/mjh/201322482
43. Tian, F. (2011). A comparison of equating / linking using the Stocking-Lord method and concurrent calibration with mixed-format tests in the non-equivalent groups common-item design under IRT [Unpublished doctoral dissertation]. Boston College.
44. Thissen, D. M., & Wainer, H. (1982). Some standard errors in item response theory. Psychometrika, 47, 397-412. https://doi.org/10.1007/BF02293705
45. Wang, T. (2006). Standard errors of equating for equipercentile equating with log-linear pre-smoothing using the delta method (Report No. 14). Center for Advanced Studies in Measurement and Assessment, Iowa.
46. Weeks, J. P. (2010). Plink: An R package for linking mixed-format tests using IRT-based methods. Journal of Statistical Software, 35(12), 1-33. https://cran.r-project.org/web/packages/plink/vignettes/plink-UD.pdf
47. Wiberg. M., & Branberg, K. (2015). Kernel equating under the non-equivalent groups with covariates design. Applied Psychological Measurement, 39(5), 349–361. https://doi.org/10.1177/0146621614567939
48. Wiberg, M., & Gonzalez, J (2016). Statistical assessment of estimated transformations in observed-score equating. Journal of Educational Measurement. 53(1), 106–125. http://www.mat.uc.cl/~jorge.gonzalez/papers/TR/Assess_TR.pdf