A Psychometric Comparison of Sato Test Theory with Classical Test Theory and Item Response Theory

Abstract

The aim of this study was to comparing the psychometric properties of the mathematics subtest items of the Determining of Students Achievement Exam and the performance / achievement / ability levels of the students who took the mathematic subtest with Sato Test Theory (STT), Classical Test Theory (CTT) and Item Response Theory (IRT) indicators. The research was conducted on 15461 8th grade students who participated in the exam in 2005. The data of this study were analyzed by examining the correlations between item discriminations, item difficulties and individual characteristics calculated in the context of different test theories. In addition, problematic test items were analyzed by clustering and observing common elements. After the analysis, it was seen that Sato Test Theory produced similar results with other theories many times in terms of determining item and individual characteristics. Moreover, some advantages related to theory were also suggested in the study. The results obtained support the claims in the literature that STT can be considered as an alternative test theory that can allow valid and reliable measurements with predictions that robust and do not contradict with other test theories.

Keywords

• Student-Problem chart analysis
• Sato test theory
• Item response theory
• test development
• psychometrics

Sato Test Kuramı’nın Klasik Test Kuramı ve Madde Tepki Kuramı ile Psikometrik Açıdan Karşılaştırılması

Öz

Bu araştırmanın amacı, Öğrenci Başarılarının Belirlenmesi Sınavı’nın (ÖBBS) matematik alt testi maddelerinin psikometrik özelliklerinin ve testi alan öğrencilerin performans/başarı/yetenek düzeylerinin Sato Test Kuramı (STK), Klasik Test Kuramı (KTK) ve Madde Tepki Kuramı (MTK) ile belirlenmesi ve elde edilen bulguların karşılaştırılarak incelenmesidir. ÖBBS’ye (2005) katılan 15461 8.sınıf öğrencisi üzerinde yürütülen bu araştırmanın verileri farklı test kuramları bağlamında hesaplanan madde ayırıcılıkları, madde güçlükleri ve birey özellikleri arasındaki korelasyonların incelenmesi, bunun yanı sıra sorunlu maddelerin kümelenerek ortak elemanlarının gözlemlenmesi şeklinde analiz edilmiştir. Yapılan analizler sonrasında STK’nın madde ve birey özelliklerinin belirlenmesi bakımından diğer kuramlarla pek çok kez benzer sonuçlar ortaya koyduğu görülmüştür. Bununla birlikte çalışmada, kurama ilişkin bazı avantajlar da öne sürülmüştür. Ulaşılan sonuçlar, alanyazında yer alan, STK’nın, diğer test kuramlarıyla çelişmeyen kestirimleriyle geçerli ve güvenilir ölçmeler yapılmasına olanak tanıyabilecek alternatif bir test kuramı olarak değerlendirilebileceği iddialarını desteklemektedir.

Anahtar Kelimeler

• Öğrenci-madde çizelgesi
• Sato test teorisi
• Madde tepki kuramı
• Test geliştirme
• Psikometri

References

1. Beavers, A.S., Lounsbury, J.W., Richards, J.K., Huck, S.W., Skolits, G.J. & Esquivel, .L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research and Evaluation, 18(6), 1-13.
2. Coughlin, K.B. (2013). An analysis of factor extraction strategies: A study of the relative strenghts of principal axis, ordinary least squares and maximum likelihood factor extraction methods in research contexts. Unpublished Doctoral Dissertation. University of South Florida, Tampa, FL, United States of America.
3. Çüm, S., Gelbal, S. & Tsai, C.P. (2016). Sato test kuramı yöntemleriyle farklı örneklemlerden elde edilen madde parametrelerinin tutarlılığının incelenmesi. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 7(1), 170-181.
4. de Ayala, R. J. (2009). The theory and practice of item response theory. New York: The Guilford Press.
5. Deng, J.L. (1982), Control problems of grey systems, Systems & Control Letters, 1 (5), 288-94.
6. Erkuş, A. (2010). Psikometrik terimlerin Türkçe karşılıklarının anlamları ile yapılan işlemlerin uyuşmazlığı. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi,1(2), 72-77.
7. Fabrigar, L. R.,Wegener,D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299.
8. Field, A. (2009). Discovering statistics using SPSS. London: Sage Publications Ltd. Hayton, J.C., Allen, D.G. & Scarpello, V. (2004). Factor retention decision in exploratoy factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205.

9. Henson, R.K. & Roberts, J.K. (2006). Use of exploratory factor analysis in published research: common errors and some comment on improved practice. Educational and Psychological Measurement, 66, 393-416.
10. Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2),179-85.
11. Hulin, C. L., Lissak, R. I., & Drasgow, F. (1982). Recovery of two and three-parameter logistic item characteristic curves: A Monte Carlo study. Applied Psychological Measurement, 6, 249-260.
12. Lin, Y.H., & Chen, S.M. (2006). The investigation of S-P chart analysis on the test evaluations of equality axiom concepts for sixth graders. Proceedings of the 2nd International Conference on Educational Technologies, Romania, Bucharest.
13. Lin, Y.H., & Yih, J.M. (2015). Application of IIRS in mathematics instruction to promote pupils decimal concept. The International Conference on Language, Education and Psychology, Taiwan.
14. Pham, D.H., Sheu, T.W., & Nagai, M. (2015). PCSP 1.0 software for partial credit S-P chart analysis. International Journal of Hybrid Information Technology. 8(6), 309-322.
15. Ree, M. J., & Jensen, H. E. (1983). Effects of sample size on linear equating of item characteristic curve parameters: Latent trait test theory and computerized adaptive testing. New York: Academic Press.
16. Sheu, T. W., Nguyen, P. T., Nguyen, P. H., Pham, D. H., Tsai, P. C., & Nagai, M. (2014b). A MATLAB toolbox for misconceptions analysis based on S-P chart, grey relational analysis and ROC. Transactions on Machine Learning and Artificial Intelligence, 2, 72-85.
17. Sheu, T.W., Nguyen, P.T., Tsai, C.P., Pham, D.H., Nguyen, P.H. ve Nagai, M. (2014c). Using grey student-problem chart in the evaluation of tests with large data sets. Education Practise and Innovation, 1(2), 2372-3106.
18. Sheu, T.W., Pham, D.H., Nguyen, P.T., & Nguyen, P.H. (2013). Amatlab toolbox for student-problem chart and grey student-problem chart and its application. International Journal of Kansei, 4(2), 75-86.
19. Sheu, T.W., Pham, D.H., Tsai, C.P., Nguyen, P.T., Nguyen, P. H. & Nagai, M. (2014a). Rasch GSP toolbox for assessing academic achievement. Journal of Software, 9(7), 1903-1913.
20. Sheu, T.W., Tsai, C.P., Tzeng, J.W., Pham, D.H., Chiang, H.J., Chang, C.L. & Nagai, M. (2013). An improved teaching strategies proposal based on student’s learning misconceptions.International Journal of Kansei Information, 4(1), 1-12.
21. Takeya, M. (1980). Construction and utilization of item relational structure graphs for use in test analysis. Japan Journal of Educational Technology, 5, 93-103.
22. Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49(1), 95-110.
23. Thompson, B. (2004). Exploratory and confirmatory factor analysis: understanding concepts and applications. Washington, DC: American Psychological Association.
24. Tsai, C.P., Sheu, T.W., Tzeng, J.W.,Chen, H.J., Chiang, H.J. & Nagai, M. (2014). Diagnose learning misconceptions based on rough sets. International Journal of Applied Mathematics and Statistics, 52(2), 63-75.
25. van der Linden, W. J. & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Springer.
26. Wang, B.T., Sheu, T.W., & Nagai, M. (2011). Evaluating the english-learning of engineering students using the grey S-P chart: a facebook case study in Taiwan. Global Journal of Engineering Education, 13(2), 51-56.
27. Wang, C.H. & Chen, C.P. (2013). Employing online S-P diagnostic table for qualitative comments on test results. The Electronic Journal of e-Learning, 11(3), 263-271.
28. Wu, H. (1999). Software Based on S-P Chart Analysis and Its Applications. Proceedings of the National Science Council, 8, 102-107.
29. Zwick,W. R. & Velicer,W. F. (1986). Factors influencing five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-44.