Validation and psychometric properties of the Computational Thinking Multidimensional Test

Year 2025, Volume: 12 Issue: 3, 825 - 842, 04.09.2025

Ali Orhan , İnan Tekin , Sedat Şen

https://doi.org/10.21449/ijate.1588902

Abstract

In this study, it was aimed to translate and adapt the Computational Thinking Multidimensional Test (CTMT) developed by Kang et al. (2023) into Turkish and to investigate its psychometric qualities with Turkish university students. Following the translation procedures of the CTMT with 12 multiple-choice questions developed based on real-life situations, the data were collected from 359 university students studying in different disciplines of a state university in the northern part of Türkiye. The structure validity and item quality of the Turkish version of CTMT were verified using multidimensional item response theory and the results showed that the Turkish test has good psychometric quality. In addition, its structure and item difficulty and discrimination characteristics are similar to the original test. Therefore, it can be stated that the Turkish version of CTMT can be used as an effective and valid assessment tool to evaluate the computational thinking skills of Turkish university students.

Keywords

Computational thinking assessment , University students , Multidimensional item response theory , Item quality analysis , Scale adaptation

Ethical Statement

Zonguldak Bülent Ecevi̇t Üni̇versi̇tesi, Human Research Ethics Committee, 24.10.2023/368263, Protocol no. 322.

References

• Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22, 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
• Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255–278. https://doi.org/10.1207/s15324818ame0704_1
• Aho, A.A. (2012). Computation and computational thinking. The Computer Journal, 55(7), 832e835. https://doi.org/10.1093/comjnl/bxs074
• Akerfeldt, A., Kjällander, S., & Petersen, P. (2024). A research review of computational thinking and programming in education. Technology, Pedagogy and Education, 33(3), 375–390. https://doi.org/10.1080/1475939X.2024.2316087
• Araujo, A.L.S.O., Andrade, W.L., Guerrero, D.D.S., & Melo, M.R.A. (2019, February). How many abilities can we measure in computational thinking? A study on bebras challenge. In E.K. Hawthorne, et al. (Eds.), SIGCSE '19 Proceedings of the 50th ACM Technical Symposium on Computer Science Education (pp. 545-551).
• Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community?. Acm Inroads, 2(1), 48-54. https://doi.org/10.1145/1929887.1929905
• Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238
• Bonifay, W. (2019). Multidimensional item response theory. Sage Publications.
• Brennan, K., & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational thinking [Conference presentation]. Annual Meeting of the American Educational Research Association, Vancouver, Canada.
• Bryne, B. (2010). Structure equation modeling with AMOS: Basic concepts applications and programs. Routledge Taylor and Francis Group.
• Buckley, S. (2012, October). The role of computational thinking and critical thinking in problem solving in a learning environment [Conference presentation]. European Conference on e-Learning, Oxfordshire, England.
• Chalmers, R.P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
• Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017a). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162 175. https://doi.org/10.1016/j.compedu.2017.03.001
• Chen, Q., Luo, W., Palardy, G.J., Glaman, R., & McEnturff, A. (2017b). The efficacy of common fit indices for enumerating classes in growth mixture models when nested data structure is ignored: A monte carlo study. Sage OPEN, 7(1), 1 19. https://doi.org/10.1177/2158244017700459
• Choi, J., Lee, Y., & Lee, E. (2017). Puzzle based algorithm learning for cultivating computational thinking. Wireless Personal Communications, 93, 131 145. https://doi.org/10.1007/s11277-016-3679-9
• Czerkawski, B.C., & Lyman, E.W. (2015). Exploring issues about computational thinking in higher education. TechTrends, 59, 57–65. https://doi.org/10.1007/s11528-015-0840-3
• Dagiene, V., Futschek, G., & Stupurienė, G. (2019). Creativity in solving short tasks for learning computational thinking. Constructivist Foundations, 14(3), 382-396.
• Dağ, F. (2019). Prepare pre-service teachers to teach computer programming skills at K-12 level: Experiences in a course. Journal of Computers in Education, 6(2), 277–313. https://doi.org/10.1007/s40692-019-00137-5
• Ding, S., Luo, F., & Tu, D. (2012). Special topic research on new progress in project response theory. Beijing Normal University Press.
• Doleck, T., Bazelais, P., Lemay, D.J., Saxena, A., & Basnet, R.B. (2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: Exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4, 355-369. https://doi.org/10.1007/s40692-017-0090-9
• Ezeamuzie, N.O., Leung, J.S., & Ting, F.S. (2022). Unleashing the potential of abstraction from cloud of computational thinking: A systematic review of literature. Journal of Educational Computing Research, 60(4), 877-905. https://doi.org/10.1177/07356331211055379
• Fernandez, J.M., Zúñiga, M.E., Rosas, M.V., & Guerrero, R.A. (2018). Experiences in learning problem-solving through computational thinking. Journal of Computer Science and Technology, 18(2), 136-142. https://doi.org/10.24215/16666038.18.e15
• Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher 42(1). 38-43. https://doi.org/10.3102/0013189X12463051
• Grover, S. (2017). Assessing algorithmic and computational thinking in K-12: Lessons from a middle school curriculum. In P. Rich & C. Hodges (Eds.), Emerging research, practice, and policy on computational thinking (pp. 269–288). Springer.
• Gülbahar, Y., Kert, S.B., & Kalelioğlu, F. (2019). The self-efficacy perception scale for computational thinking skill: Validity and reliability study. Turkish Journal of Computer and Mathematics Education, 10(1), 1–29.
• Hanid, M.F.A., Said, M.N.H.M., Yahaya, N., & Abdullah, Z. (2022). The elements of computational thinking in learning geometry by using augmented reality application. International Journal of Interactive Mobile Technologies, 16(2), 28 41. https://doi.org/10.3991/ijim.v16i02.27295
• Hershkovitz, A., Sitman, R., Israel-Fishelson, R., Eguíluz, A., Garaizar, P., & Guenaga, M. (2019). Creativity in the acquisition of computational thinking. Interactive Learning Environments, 27(5-6), 628-644. https://doi.org/10.1080/10494820.2019.1610451
• He, Z., Wu, X., Wang, Q., & Huang, C. (2021). Developing eighth-grade students’ computational thinking with critical reflection. Sustainability, 13, 1 21. https://doi.org/10.3390/su132011192
• Hsu, T.C., Chang, S.C., & Hung, Y.T. (2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers & Education, 126, 296–310.
• International Society for Technology in Education (ISTE). (2024, June 20). Computational thinking competencies. https://iste.org/standards/computational-thinking-competencies
• International Test Commission (2017, December 25). The ITC guidelines for translating and adapting tests (Second edition). https://www.intestcom.org/files/guideline_test_adaptation_2ed.pdf
• Israel-Fishelson, R., Hershkovitz, A., Eguíluz, A., Garaizar, P., & Guenaga, M. (2021). The associations between computational thinking and creativity: The role of personal characteristics. Journal of Educational Computing Research, 58(8), 1415 1447. https://doi.org/10.1177/0735633120940954
• Jiang, B., & Li, Z. (2021). Effect of Scratch on computational thinking skills of Chinese primary school students. Journal of Computers in Education, 8(4), 505 525. https://doi.org/10.1007/s40692-021-00190-z
• Kang, C., Liu, N., Zhu, Y., Li, F., & Zeng, P. (2023). Developing college students’ computational thinking multidimensional test based on life story situations. Education and Information Technologies, 28(3), 2661-2679. https://doi.org/10.1007/s10639-022-11189-z
• Kılıç, S., Gökoğlu, S., & Öztürk, M. (2021). A valid and reliable scale for developing programming-oriented computational thinking. Journal of Educational Computing Research, 59(2), 257-286. https://doi.org/10.1177/0735633120964402
• Knuth, D.E. (1974). Computer science and its relation to mathematics. The American Mathematical Monthly, 81(4), 323-343. https://doi.org/10.2307/2318994
• Korkmaz, Ö., Çakır, R., & Özden, M.Y. (2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558-569. https://doi.org/10.1016/j.chb.2017.01.005
• Kukul, V., & Karatas, S. (2019). Computational thinking self-efficacy scale: Development, validity and reliability. Informatics in Education, 18(1), 151 164. https://doi.org/10.15388/infedu.2019.07
• Kules, B. (2016). Computational thinking is critical thinking: Connecting to university discourse, goals, and learning outcomes. Proceedings of the Association for Information Science and Technology, 53(1), 1–16.
• Küçükaydın, M., & Akkanat, Ç. (2022). Adaptation into Turkish of the Computational Thinking Test for Primary School Students. Problems of Education in the 21st Century, 80(6), 765-776. https://doi.org/10.33225/pec/22.80.765
• Lai, R.P.Y. (2021). Beyond programming: A computer-based assessment of computational thinking competency. ACM Transactions on Computing Education, 22(2), 14–27. https://doi.org/10.1145/3486598
• Lai, R.P.Y., & Ellefson, M.R. (2023). How multidimensional is computational thinking competency? A bi-factor model of the computational thinking challenge. Journal of Educational Computing Research, 61(2), 259–282. https://doi.org/10.31219/osf.io/e8n3h
• Lee, M.G. (2019, May 30). Teaching computational thinking in early elementary. https://csteachers.org/teaching-computational-thinking-in-early-elementary/
• Lei, H., Chiu, M.M., Li, F., Wang, X., & Geng, Y.J. (2020). Computational thinking and academic achievement: A meta-analysis among students. Children and Youth Services Review, 118, 1-8. https://doi.org/10.1016/j.childyouth.2020.105439
• Lemay, D.J., Basnet, R.B., Doleck, T., Bazelais, P., & Saxena, A. (2021). Instructional interventions for computational thinking: Examining the link between computational thinking and academic performance. Computers and Education Open, 2, 1 6. https://doi.org/10.1016/j.caeo.2021.100056
• Li, Y., Schoenfeld, A.H., DiSessa, A.A., Graesser, A.C., Benson, L.C., English, L.D., & Duschl, R.A. (2020). Computational thinking is more about thinking than computing. Journal for STEM Education Research, 3, 1–18. https://doi.org/10.1007/s41979-020-00030-2
• Lu, C., Macdonald, R., Odell, B., Kokhan, V., Epp, D.C., & Cutumisu, M. (2022). A scoping review of computational thinking assessments in higher education. Journal of Computing in Higher Education, 34, 416-461. https://doi.org/10.1007/s12528-021-09305-y
• Lye, S.Y., Koh, J.H.L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12?. Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
• Maydeu-Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71(4), 713 732. https://doi.org/10.1007/s11336-005-1295-9
• McDonald, R.P. (2000). A basis for multidimensional item response theory. Applied Psychological Measurement, 24(2), 99-114. https://doi.org/10.1177/01466210022031552
• Peel, A., & Friedrichsen, P. (2018). Algorithms, abstractions, and iterations: Teaching computational thinking using protein synthesis translation. The American Biology Teacher, 80(1), 21-28. https://doi.org/10.1525/abt.2018.80.1.21
• Rao, T.S.S., & Bhagat, K.K. (2024). Computational thinking for the digital age: a systematic review of tools, pedagogical strategies, and assessment practices. Education Technology Research and Development, 72, 1893–1924. https://doi.org/10.1007/s11423-024-10364-y
• Reckase, M.D. (2006). Multidimensional item response theory. In C.R. Rao & S. Sinharay (Eds), Handbook of statistics (pp. 607-642). Elsevier.
• Reckase, M.D. (2009). Multidimensional item response theory (1st ed.). Springer.
• Reckase, M.D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25- 36. https://doi.org/10.1177/0146621697211002
• Rode, J.A., Weibert, A., Marshall, A., Aal, K., von Rekowski, T., Elmimouni, H., & Booker, J. (2015, September). From computational thinking to computational making [Conference presentation]. ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan.
• Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(42), 1-15. https://doi.org/10.1186/s41239-017-0080-z
• Schober, P., Boer, C., & Schwarte, L.A. (2018). Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia, 126(5), 1763 1768. https://doi.org/10.1213/ANE.0000000000002864
• Selby, C.C. (2015, November). Relationships: computational thinking, pedagogy of programming, and Bloom's Taxonomy. Proceedings of the workshop in primary and secondary computing education, 80-87.
• Standl, B. (2016, April). A case study on cooperative problem solving processes in small 9th grade student groups. Proceedings of IEEE Global Engineering Education Conference, 961-967. IEEE.
• Tikva, C., & Tambouris, E. (2021). A systematic mapping study on teaching and learning Computational Thinking through programming in higher education. Thinking Skills and Creativity, 41(100849), 1-18. https://doi.org/10.1016/j.tsc.2021.100849
• Topallı, D., & Cağıltay, N.E. (2018). Improving programming skills in engineering education through problem-based game projects with scratch. Computers & Education, 120, 64–74. https://doi.org/10.1016/j.compedu.2018.01.011
• Tran, Y. (2019). Computational thinking equity in elementary classrooms: What third-grade students know and can do. Journal of Educational Computing Research, 57(1), 3-31. https://doi.org/10.1177/0735633117743918
• Tsai, M.J., Liang, J.C., & Hsu, C.Y. (2021). The computational thinking scale for computer literacy education. Journal of Educational Computing Research, 59(4), 579 602. https://doi.org/10.1177/0735633120972356
• Vrieze, S.I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243. https://doi.org/10.1037/a0027127
• Wing, J.M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366 (1881), 3717-3725.
• Wing, J.M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
• Voskoglou, M.G. & Buckley, S. (2012). Problem solving and computers in a learning environment. Egyptian Computer Science Journal, 36(4), 28-46.
• Yağcı, M. (2019). A valid and reliable tool for examining computational thinking skills. Education and Information Technologies, 24(1), 929–951. https://doi.org/10.1007/s10639-018-9801-8