Ağaç Temelli ve Birleştirilmiş Ağaç Temelli Bilgisayarda Bireyselleştirilmiş Test Yöntemlerinin Parametrelerine Dayalı Karşılaştırmalı Bir İnceleme

Öz

Bu araştırmada, R programlama dili kullanılarak Birleştirilmiş Ağaç Temelli Bilgisayarda Bireyselleştirilmiş Test (Birleştirilmiş Ağaç BBT) yöntemine ait parametreler belirlenmiştir. Çalışmada, Birleştirilmiş Ağaç BBT ile Ağaç BBT’nin performansı farklı koşullar altında karşılaştırılmıştır. Test etkililiği ve ölçme kesinliği; seviye başına maksimum düğüm sayısı, birleştirilecek iki düğümün yoğunluk fonksiyonlarının minimum kesişimi, madde havuzundaki madde sayısı, her bireye uygulanacak maksimum madde sayısı ve maddenin maruz kalma oranına göre incelenmiştir. Araştırma, gerçek veri seti kullanılarak yürütülmüştür. Madde havuzunda 919 bireye uygulanmış 256 madde bulunmaktadır. Tüm koşullarda 25 tekrarın ortalaması alınmıştır. Bulgular, madde havuzundaki madde sayısı arttıkça Birleştirilmiş Ağaç BBT’lerde yetenek kestiriminde daha düşük hata değerleri elde edildiğini göstermiştir. Ayrıca, seviye başına maksimum düğüm sayısı, dal sayısı ve minimum benzerlik gibi parametrelerle birleştirme parametreleri arttıkça test etkililiği ve ölçme kesinliğinin yükseldiği belirlenmiştir. Bununla birlikte, Birleştirilmiş Ağaç BBT için seviye başına maksimum dal sayısının belirleyici bir parametre olduğu, her bir birey için kullanılacak maksimum madde sayısının ise belirleyici olmadığı anlaşılmıştır.

Anahtar Kelimeler

• birleştirilmiş ağaç tabanlı bilgisayarda bireyselleştirilmiş test
• maksimum düğüm sayısı
• madde maruz kalma oranı
• maksimum dal sayısı
• birleştirme parametreleri

A Comparative Review of the Parameters of Tree-Based and Marged Tree-Based Computerized Adaptive Testing Methods

Öz

In this study, parameters of the Merged Tree-Based Computerized Adaptive Test (Merged Tree CAT) method were determined using the R programming language. The performance of the Merged Tree CAT was compared with that of the Tree CAT under different conditions. Test efficiency and measurement precision were examined with respect to the maximum number of nodes per ability level, the minimum intersection of the density functions of the two nodes to be merged, the number of items in the item pool, the maximum number of items administered per examinee, and the item exposure rate. The study was conducted using a real dataset comprising 256 items administered to 919 examinees. For all conditions, the average of 25 replications was taken. The findings indicated that as the number of items in the item pool increased, lower error values in ability estimation were obtained in the Merged Tree CAT. Furthermore, as the merging parameters such as the maximum number of nodes per level, the number of branches, and minimum similarity increased, test efficiency and measurement precision also improved. However, it was observed that the maximum number of branches per ability level was a determining parameter for the Merged Tree CAT, whereas the maximum number of items administered per examinee was not a critical factor.

Anahtar Kelimeler

• marged testing on a fused tree-based computer
• maximum number of nodes
• item exposure rate
• maximum number of branches
• merge parameters

Kaynakça

1. Armstrong, R. D., & Edmonds, R. (2004). Online computerized adaptive testing overview. J. M. P. Fernández-Ballesteros (Haz.), Encyclopedia of psychological assessment içinde (s. 706–711). Sage.
2. Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444. https://doi.org/10.1177/014662168200600405
3. Cha, S.-H. (2007). Comprehensive survey on distance/similarity measures between probability density functions. International Journal of Mathematical Models and Methods in Applied Sciences, 1(4), 300–307.
4. Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
5. Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229. https://doi.org/10.1177/014662169602000303
6. Delgado-Gómez, D., Laria, J. C., & Ruiz-Hernández, D. (2019). Computerized adaptive test and decision trees: A unifying approach. Expert Systems with Applications, 117, 358–366. https://doi.org/10.1016/j.eswa.2018.09.052
7. Delgado-Gómez, D., Morales-Álvarez, P., & Martínez-Ramón, M. (2019). A tree-based method for adaptive testing. IEEE Transactions on Learning Technologies, 12(1), 52–62. https://doi.org/10.1109/TLT.2018.2794998
8. Georgiadou, E., Triantafillou, E., & Economides, A. A. (2007). A review of item exposure control strategies for computerized adaptive testing developed from 1983 to 2005. The Journal of Technology, Learning and Assessment, 5(8), 4–38. https://ejournals.bc.edu/index.php/jtla/article/view/1645

9. Kaptan, S. (1995). Araştırma yöntem ve teknikleri. Anadolu Üniversitesi Yayınları.
10. Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359–375. https://doi.org/10.1207/s15324818ame0204_6
11. Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices (2. baskı). Springer.
12. Liao, C. H., & Lin, C. H. (2022). Enhanced item selection strategies in multidimensional CAT: A decision-tree-based simulation study. Applied Psychological Measurement, 46(6), 455–472. https://doi.org/10.1177/01466216221080424
13. Lord, F. M. (2012). Applications of item response theory to practical testing problems. Routledge.
14. Lu, P., Zhou, D., Qin, S., Cong, X., & Zhong, S. (2012). The study of item selection method in CAT. Z. Li, X. Li, Y. Liu & Z. Cai, (Haz.) Computational intelligence and intelligent systems içinde (s. 403–415). Springer. https://doi.org/10.1007/978-3-642-34289-9_45
15. Magis, D. (2015). A note on the equivalence between observed and expected information functions with polytomous IRT models. Journal of Educational and Behavioral Statistics, 40(1), 96–105. https://doi.org/10.3102/1076998614558122
16. Magis, D., & Barrada, J. R. (2017). Computerized adaptive testing with R: Recent updates of the package catR. Journal of Statistical Software, 76(1), 1–19. https://doi.org/10.18637/jss.v076.c015
17. Miller, I., & Miller, M. (2004). John E. Freund’s mathematical statistics with applications (7. baskı). Pearson Education.
18. Nydick, J., & Weiss, D. J. (2009). Linking multidimensional CAT item pools. D. J. Weiss (Haz.), Proceedings of the 2009 GMAC conference on computerized adaptive testing içinde (s.1–13). https://www.psych.umn.edu/psylabs/CATCentral/
19. Parshall, C. G., Spray, J. A., Kalohn, J. C., & Davey, T. (2002). Practical considerations in computer-based testing. Springer.
20. R Core Team (2013). R: A language and environment for statistical computing, (Version 3.0.1), Vienna, Austria: R Foundation for Statistical Computing. http://www.R-project.org/
21. Reckase, M. D. (1981). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 5(4), 401–412. https://doi.org/10.1177/014662168100500401
22. Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35(4), 311–327. https://doi.org/10.1111/j.1745-3984.1998.tb00541.x
23. Rodríguez-Cuadrado, J., Delgado-Gómez, D., Laria, J. C., & Ruiz-Hernández, D. (2020). Merged TreeCAT: A fast method for building precise computerized adaptive tests based on decision trees. Expert Systems with Applications, 143, 113066. https://doi.org/10.1016/j.eswa.2019.113066
24. Rodríguez-Cuadrado, J., Delgado-Gómez, D., & Ruiz-Hernández, D. (2023). A hybrid decision-tree approach for large-scale adaptive testing: Advances over classic CAT. Expert Systems with Applications, 226, 120276. https://doi.org/10.1016/j.eswa.2023.120276
25. Rodríguez-Cuadrado, S., García-Pérez, M. Á., & Delgado-Gómez, D. (2021). Combining decision trees with item response theory for computerized adaptive testing: The Joint Tree-CAT. Applied Psychological Measurement, 45(3), 183–199. https://doi.org/10.1177/0146621620941960
26. Shin, C. D. (2017). Conditional randomesque method for item exposure control in CAT. International Journal of Intelligent Technologies & Applied Statistics, 10(3), 153–165. https://doi.org/10.6148/IJITAS.2017.1003.02
27. Shin, C. D., & Kim, H. (2021). Adaptive testing based on ensemble trees: Performance comparison under item exposure constraints. Educational and Psychological Measurement, 81(4), 682–702. https://doi.org/10.1177/0013164420975192
28. Sympson, J. B., & Hetter, R. D. (1985, Ekim). Controlling item-exposure rates in computerized adaptive testing. Paper presented at the 27th Annual Meeting of the Military Testing Association, San Diego, CA, United States. Proceedings of the Military Testing Association, (s. 973–977). Navy Personnel Research and Development Center. Retrieved from https://www.iacat.org/content/controlling-item-exposure-rates-computerizedadaptive-testing
29. Ueno, M., & Songmuang, P. (2010). Computerized adaptive testing based on decision tree. 2010 10th IEEE International Conference on Advanced Learning Technologies içinde (s. 191–193). IEEE. https://doi.org/10.1109/ICALT.2010.58
30. Van der Linden, W. J. (2003). Principles of adaptive testing. W. J. van der Linden & C. A. W. Glas (Haz.), Computerized adaptive testing: Theory and practice içinde (s. 1–26). Springer. https://doi.org/10.1007/978-1-4757-3920-9_1
31. Van der Linden, W. J., & Pashley, P. J. (2009). Item selection and ability estimation in adaptive testing. W. J. van der Linden & C. A. W. Glas (Haz.), Elements of adaptive testing içinde (s. 3–30). Springer. https://doi.org/10.1007/978-0-387-85461-8_1
32. Van der Linden, W. J., & Veldkamp, B. P. (2004). Constraining item exposure in computerized adaptive testing with shadow tests. Journal of Educational and Behavioral Statistics, 29(3), 273–291. https://doi.org/10.3102/10769986029003293
33. Van der Linden, W. J., & Veldkamp, B. P. (2005). Constraining item exposure in computerized adaptive testing with shadow tests (LSAC Research Report No. 02-03). Law School Admission Council.
34. Veerkamp, W. J., & Berger, M. P. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22(2), 203–226. https://doi.org/10.3102/10769986022002203
35. Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473–492. https://doi.org/10.1177/014662168200600405
36. Weiss, D. J. (2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37(2), 70–84.
37. Yan, D., Lewis, C., & Stocking, M. (2004). Adaptive testing with regression trees in the presence of multidimensionality. Journal of Educational and Behavioral Statistics, 29(3), 293–316. https://doi.org/10.3102/10769986029003293