Comparative Study of Classical Test Theory and Item Response Theory Using Diagnostic Quantitative Economics Skill Test Item Analysis Results

Abstract

In this study a comparison was made on DQEST item parameters and test statistics results estimated using Classical Test Theory (CTT) approach and Item Response Theory (IRT) three parameter logistic model (3PLM) to find out the similarities and differences in the two frameworks. 517 randomly selected senior secondary three (SS3) economics students comprised the sample. Three research questions guided the study. Responses obtained from SS3 economics students in 50 multiple choice items of Diagnostic Quantitative Economics Skill Test (DQEST) were used for the analysis. DQEST items certified the unidimensionality, local independence and model-data fit assumptions. Then results from CTT and IRT analyses were compared. In terms of very difficult item and item that discriminate poorly, CTT were found not to be comparable with the 3PLM the most appropriate model for DQEST data. The calculated reliability value for CTT was found to be low when compared to that generated by 3PLM. Therefore, it could be concluded that there was disparity between CTT approach and 3 parameter IRT model in terms of item parameters and test statistics. Thus IRT model with the best data fit should be employed for an enhanced test validity and reliability.

Keywords

• Item response theory,classical test theory,item parameters,local independence,unidimensionality

References

1. Adedoyin, C. (2010). Investigating the invariance of persons’ parameter estimates based on classical test and item response theories. An International Journal on Education Science, 2(2), 107-113. Retrieved from http:// krepublishers.com/...Journals/...2...2...Adedoyin.../IJES-2-2-107-10-033-Adedoyin-O...
2. Adedoyin, O.O., & Mokobi, T. (2013). Using IRT psychometric analysis in examining the quality of junior certificate mathematics multiple choice examination test items. International Journal of Asian Social Science, 3(4), 992-1011. Retrieved from http://www.aessweb.com/journal etail.php?id=5007
3. Ani, E.N. (2014). Application of item response theory in the development and validation of multiple choice test in economics. (Master’s thesis). University of Nigeria, Nsukka.
4. Bichi, A.A., Embong, R., Mamat, M., & Maiwada, D. A. (2015). Comparison of classical test theory and item response theory: a review of empirical studies. Australian Journal of Basic and Applied Sciences, 9(7), 549-556. Doi:10.13140/RG.2.1.1561.
5. Cappelleri, J.C., Lundy, J.J., & Hays, R. D. (2014). Overview of classical test theory and item response theory for quantitative assessment of items in developing patient-reported outcome measures. Doi: 10.1016/j.clinthera.2014.04.006
6. Esomonu, N.P.M. & Eleje, L.I. (2017). Diagnostic quantitative economics skill test for secondary schools: Development and validation using item response theory. Journal of Education and Practice, 8(22), 110-125. Retrieved from www.iiste.org
7. Fan, X. (1998). Item response theory and classical test theory: an empirical comparison of their item/person statistics. Educational and Psychological Measurement, 58(3), 357. Retrieved from http://journals.sagepub.com/doi/abs/10.1177/0013164498058003001
8. Guler, N., Uynik, G.K., & Teker, G.T. (2014). Comparison of classical test theory and item response theory in terms of item parameters. European Journal of Research on Education, 2(1), 1-6. Retrieved from http://iassr.org/journal

9. Hambleton, R.K. and Swaminathan, H. (1985). Item response theory. Principles and application. Retrieved from www.springer.com/gp/book/9780898380651
10. Hambleton, R.K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA:Sage Publications
11. Hays, R. D., Bjorner, J. B., Revicki, D. A., Spritzer, K. L., & Cella, D. (2009). Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Quality of Life Research, (18), 873–80.
12. Kolawole, E.B. (2010). Principles of test construction and administration (Revised Edition). Lagos: Bolabay Publications.
13. Kyung, T. H. (2013). Windows software that generates IRT parameters and item responses: research and evaluation program methods (REMP). University of Massachusetts Amherst. Retrieved from https://www.umass.edu/remp/software/simcata/wingen/homeF.html
14. Lord, F.M. (1980). Application of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
15. Magis, D. (2007). Influence, information and item response theory in discrete data analysis. Retrieved from http://bictel.ulg.ac.be/ETDdb/collection/available/ULgetd-06122007-100147/.
16. Mead, A.D., & Meade, A.W. (2010). Item selection using CTT and IRT with unrepresentative samples. Paper presented at the twenty-fifth annual meeting of the Society for Industrial and Organizational Psychology in Atlanta, GA. Retrieved from https://www.researchgate.net/...Classical_Test_Theory...Item.../Comparison-of-Classica...
17. Ojerinde, D. (2013). Classical test theory (CTT) vs item response theory (IRT): an evaluation of the comparability of item analysis results. Retrieved from https://ui.edu.ng/sites/.../PROF%20OJERINDE'S%20LECTURE%20(Autosaved).pdf
18. Ojerinde, D., & Ifewulu, B. C. (2012). Item unidimensionality using 2010 unified tertiary matriculation examination mathematics pre-test. A paper presented at the 2012 international conference of IAEA, Kazastan. Retrieved from https://ui.edu.ng/sites/.../PROF%20OJERINDE'S%20LECTURE%20(Autosaved).pdf
19. Rijn, R.W.V., Sinharay, S., Haberman, S.J., & Johnson, M.S. (2016). Assessment of fit of item response theory models used in large-scale educational survey assessments. DOI: 10.1186/s40536-016-0025-3
20. Stage, C. (1998). A comparison between item analysis based on item response theory and classical test theory. A study of the SweSAT subtest ERC. (Educational Measurement). Umeå University, Department of Educational Measurement. Retrieved from www.edusci.umu.se/digitalAssets/60/60608_enr3098sec.pdf
21. Thorpe, G. L., & Favia, A. (2012). Data analysis using item response theory methodology: an introduction to selected programs and applications. Retrieved from http://digitalcommons.lidrary.umaine.edu/psy_facpub/20