Validation and psychometric properties of the Computational Thinking Multidimensional Test

Abstract

In this study, it was aimed to translate and adapt the Computational Thinking Multidimensional Test (CTMT) developed by Kang et al. (2023) into Turkish and to investigate its psychometric qualities with Turkish university students. Following the translation procedures of the CTMT with 12 multiple-choice questions developed based on real-life situations, the data were collected from 359 university students studying in different disciplines of a state university in the northern part of Türkiye. The structure validity and item quality of the Turkish version of CTMT were verified using multidimensional item response theory and the results showed that the Turkish test has good psychometric quality. In addition, its structure and item difficulty and discrimination characteristics are similar to the original test. Therefore, it can be stated that the Turkish version of CTMT can be used as an effective and valid assessment tool to evaluate the computational thinking skills of Turkish university students.

Keywords

• Computational thinking assessment
• University students
• Multidimensional item response theory
• Item quality analysis
• Scale adaptation

References

1. Ackerman, T.A., Gierl, M.J., & Walker, C.M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22, 37-51. https://doi.org/10.1111/j.1745-3992.2003.tb00136.x
2. Ackerman, T.A. (1994). Using multidimensional item response theory to understand what items and tests are measuring. Applied Measurement in Education, 7(4), 255–278. https://doi.org/10.1207/s15324818ame0704_1
3. Aho, A.A. (2012). Computation and computational thinking. The Computer Journal, 55(7), 832e835. https://doi.org/10.1093/comjnl/bxs074
4. Akerfeldt, A., Kjällander, S., & Petersen, P. (2024). A research review of computational thinking and programming in education. Technology, Pedagogy and Education, 33(3), 375–390. https://doi.org/10.1080/1475939X.2024.2316087
5. Araujo, A.L.S.O., Andrade, W.L., Guerrero, D.D.S., & Melo, M.R.A. (2019, February). How many abilities can we measure in computational thinking? A study on bebras challenge. In E.K. Hawthorne, et al. (Eds.), SIGCSE '19 Proceedings of the 50th ACM Technical Symposium on Computer Science Education (pp. 545-551).
6. Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community?. Acm Inroads, 2(1), 48-54. https://doi.org/10.1145/1929887.1929905
7. Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238
8. Bonifay, W. (2019). Multidimensional item response theory. Sage Publications.

9. Brennan, K., & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational thinking [Conference presentation]. Annual Meeting of the American Educational Research Association, Vancouver, Canada.
10. Bryne, B. (2010). Structure equation modeling with AMOS: Basic concepts applications and programs. Routledge Taylor and Francis Group.
11. Buckley, S. (2012, October). The role of computational thinking and critical thinking in problem solving in a learning environment [Conference presentation]. European Conference on e-Learning, Oxfordshire, England.
12. Chalmers, R.P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
13. Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017a). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162 175. https://doi.org/10.1016/j.compedu.2017.03.001
14. Chen, Q., Luo, W., Palardy, G.J., Glaman, R., & McEnturff, A. (2017b). The efficacy of common fit indices for enumerating classes in growth mixture models when nested data structure is ignored: A monte carlo study. Sage OPEN, 7(1), 1 19. https://doi.org/10.1177/2158244017700459
15. Choi, J., Lee, Y., & Lee, E. (2017). Puzzle based algorithm learning for cultivating computational thinking. Wireless Personal Communications, 93, 131 145. https://doi.org/10.1007/s11277-016-3679-9
16. Czerkawski, B.C., & Lyman, E.W. (2015). Exploring issues about computational thinking in higher education. TechTrends, 59, 57–65. https://doi.org/10.1007/s11528-015-0840-3
17. Dagiene, V., Futschek, G., & Stupurienė, G. (2019). Creativity in solving short tasks for learning computational thinking. Constructivist Foundations, 14(3), 382-396.
18. Dağ, F. (2019). Prepare pre-service teachers to teach computer programming skills at K-12 level: Experiences in a course. Journal of Computers in Education, 6(2), 277–313. https://doi.org/10.1007/s40692-019-00137-5
19. Ding, S., Luo, F., & Tu, D. (2012). Special topic research on new progress in project response theory. Beijing Normal University Press.
20. Doleck, T., Bazelais, P., Lemay, D.J., Saxena, A., & Basnet, R.B. (2017). Algorithmic thinking, cooperativity, creativity, critical thinking, and problem solving: Exploring the relationship between computational thinking skills and academic performance. Journal of Computers in Education, 4, 355-369. https://doi.org/10.1007/s40692-017-0090-9
21. Ezeamuzie, N.O., Leung, J.S., & Ting, F.S. (2022). Unleashing the potential of abstraction from cloud of computational thinking: A systematic review of literature. Journal of Educational Computing Research, 60(4), 877-905. https://doi.org/10.1177/07356331211055379
22. Fernandez, J.M., Zúñiga, M.E., Rosas, M.V., & Guerrero, R.A. (2018). Experiences in learning problem-solving through computational thinking. Journal of Computer Science and Technology, 18(2), 136-142. https://doi.org/10.24215/16666038.18.e15
23. Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher 42(1). 38-43. https://doi.org/10.3102/0013189X12463051
24. Grover, S. (2017). Assessing algorithmic and computational thinking in K-12: Lessons from a middle school curriculum. In P. Rich & C. Hodges (Eds.), Emerging research, practice, and policy on computational thinking (pp. 269–288). Springer.
25. Gülbahar, Y., Kert, S.B., & Kalelioğlu, F. (2019). The self-efficacy perception scale for computational thinking skill: Validity and reliability study. Turkish Journal of Computer and Mathematics Education, 10(1), 1–29.
26. Hanid, M.F.A., Said, M.N.H.M., Yahaya, N., & Abdullah, Z. (2022). The elements of computational thinking in learning geometry by using augmented reality application. International Journal of Interactive Mobile Technologies, 16(2), 28 41. https://doi.org/10.3991/ijim.v16i02.27295
27. Hershkovitz, A., Sitman, R., Israel-Fishelson, R., Eguíluz, A., Garaizar, P., & Guenaga, M. (2019). Creativity in the acquisition of computational thinking. Interactive Learning Environments, 27(5-6), 628-644. https://doi.org/10.1080/10494820.2019.1610451
28. He, Z., Wu, X., Wang, Q., & Huang, C. (2021). Developing eighth-grade students’ computational thinking with critical reflection. Sustainability, 13, 1 21. https://doi.org/10.3390/su132011192
29. Hsu, T.C., Chang, S.C., & Hung, Y.T. (2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers & Education, 126, 296–310.
30. International Society for Technology in Education (ISTE). (2024, June 20). Computational thinking competencies. https://iste.org/standards/computational-thinking-competencies
31. International Test Commission (2017, December 25). The ITC guidelines for translating and adapting tests (Second edition). https://www.intestcom.org/files/guideline_test_adaptation_2ed.pdf
32. Israel-Fishelson, R., Hershkovitz, A., Eguíluz, A., Garaizar, P., & Guenaga, M. (2021). The associations between computational thinking and creativity: The role of personal characteristics. Journal of Educational Computing Research, 58(8), 1415 1447. https://doi.org/10.1177/0735633120940954
33. Jiang, B., & Li, Z. (2021). Effect of Scratch on computational thinking skills of Chinese primary school students. Journal of Computers in Education, 8(4), 505 525. https://doi.org/10.1007/s40692-021-00190-z
34. Kang, C., Liu, N., Zhu, Y., Li, F., & Zeng, P. (2023). Developing college students’ computational thinking multidimensional test based on life story situations. Education and Information Technologies, 28(3), 2661-2679. https://doi.org/10.1007/s10639-022-11189-z
35. Kılıç, S., Gökoğlu, S., & Öztürk, M. (2021). A valid and reliable scale for developing programming-oriented computational thinking. Journal of Educational Computing Research, 59(2), 257-286. https://doi.org/10.1177/0735633120964402
36. Knuth, D.E. (1974). Computer science and its relation to mathematics. The American Mathematical Monthly, 81(4), 323-343. https://doi.org/10.2307/2318994
37. Korkmaz, Ö., Çakır, R., & Özden, M.Y. (2017). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558-569. https://doi.org/10.1016/j.chb.2017.01.005
38. Kukul, V., & Karatas, S. (2019). Computational thinking self-efficacy scale: Development, validity and reliability. Informatics in Education, 18(1), 151 164. https://doi.org/10.15388/infedu.2019.07
39. Kules, B. (2016). Computational thinking is critical thinking: Connecting to university discourse, goals, and learning outcomes. Proceedings of the Association for Information Science and Technology, 53(1), 1–16.
40. Küçükaydın, M., & Akkanat, Ç. (2022). Adaptation into Turkish of the Computational Thinking Test for Primary School Students. Problems of Education in the 21st Century, 80(6), 765-776. https://doi.org/10.33225/pec/22.80.765
41. Lai, R.P.Y. (2021). Beyond programming: A computer-based assessment of computational thinking competency. ACM Transactions on Computing Education, 22(2), 14–27. https://doi.org/10.1145/3486598
42. Lai, R.P.Y., & Ellefson, M.R. (2023). How multidimensional is computational thinking competency? A bi-factor model of the computational thinking challenge. Journal of Educational Computing Research, 61(2), 259–282. https://doi.org/10.31219/osf.io/e8n3h
43. Lee, M.G. (2019, May 30). Teaching computational thinking in early elementary. https://csteachers.org/teaching-computational-thinking-in-early-elementary/
44. Lei, H., Chiu, M.M., Li, F., Wang, X., & Geng, Y.J. (2020). Computational thinking and academic achievement: A meta-analysis among students. Children and Youth Services Review, 118, 1-8. https://doi.org/10.1016/j.childyouth.2020.105439
45. Lemay, D.J., Basnet, R.B., Doleck, T., Bazelais, P., & Saxena, A. (2021). Instructional interventions for computational thinking: Examining the link between computational thinking and academic performance. Computers and Education Open, 2, 1 6. https://doi.org/10.1016/j.caeo.2021.100056
46. Li, Y., Schoenfeld, A.H., DiSessa, A.A., Graesser, A.C., Benson, L.C., English, L.D., & Duschl, R.A. (2020). Computational thinking is more about thinking than computing. Journal for STEM Education Research, 3, 1–18. https://doi.org/10.1007/s41979-020-00030-2
47. Lu, C., Macdonald, R., Odell, B., Kokhan, V., Epp, D.C., & Cutumisu, M. (2022). A scoping review of computational thinking assessments in higher education. Journal of Computing in Higher Education, 34, 416-461. https://doi.org/10.1007/s12528-021-09305-y
48. Lye, S.Y., Koh, J.H.L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12?. Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
49. Maydeu-Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71(4), 713 732. https://doi.org/10.1007/s11336-005-1295-9
50. McDonald, R.P. (2000). A basis for multidimensional item response theory. Applied Psychological Measurement, 24(2), 99-114. https://doi.org/10.1177/01466210022031552
51. Peel, A., & Friedrichsen, P. (2018). Algorithms, abstractions, and iterations: Teaching computational thinking using protein synthesis translation. The American Biology Teacher, 80(1), 21-28. https://doi.org/10.1525/abt.2018.80.1.21
52. Rao, T.S.S., & Bhagat, K.K. (2024). Computational thinking for the digital age: a systematic review of tools, pedagogical strategies, and assessment practices. Education Technology Research and Development, 72, 1893–1924. https://doi.org/10.1007/s11423-024-10364-y
53. Reckase, M.D. (2006). Multidimensional item response theory. In C.R. Rao & S. Sinharay (Eds), Handbook of statistics (pp. 607-642). Elsevier.
54. Reckase, M.D. (2009). Multidimensional item response theory (1st ed.). Springer.
55. Reckase, M.D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25- 36. https://doi.org/10.1177/0146621697211002
56. Rode, J.A., Weibert, A., Marshall, A., Aal, K., von Rekowski, T., Elmimouni, H., & Booker, J. (2015, September). From computational thinking to computational making [Conference presentation]. ACM International Joint Conference on Pervasive and Ubiquitous Computing, Osaka, Japan.
57. Romero, M., Lepage, A., & Lille, B. (2017). Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 14(42), 1-15. https://doi.org/10.1186/s41239-017-0080-z
58. Schober, P., Boer, C., & Schwarte, L.A. (2018). Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia, 126(5), 1763 1768. https://doi.org/10.1213/ANE.0000000000002864
59. Selby, C.C. (2015, November). Relationships: computational thinking, pedagogy of programming, and Bloom's Taxonomy. Proceedings of the workshop in primary and secondary computing education, 80-87.
60. Standl, B. (2016, April). A case study on cooperative problem solving processes in small 9th grade student groups. Proceedings of IEEE Global Engineering Education Conference, 961-967. IEEE.
61. Tikva, C., & Tambouris, E. (2021). A systematic mapping study on teaching and learning Computational Thinking through programming in higher education. Thinking Skills and Creativity, 41(100849), 1-18. https://doi.org/10.1016/j.tsc.2021.100849
62. Topallı, D., & Cağıltay, N.E. (2018). Improving programming skills in engineering education through problem-based game projects with scratch. Computers & Education, 120, 64–74. https://doi.org/10.1016/j.compedu.2018.01.011
63. Tran, Y. (2019). Computational thinking equity in elementary classrooms: What third-grade students know and can do. Journal of Educational Computing Research, 57(1), 3-31. https://doi.org/10.1177/0735633117743918
64. Tsai, M.J., Liang, J.C., & Hsu, C.Y. (2021). The computational thinking scale for computer literacy education. Journal of Educational Computing Research, 59(4), 579 602. https://doi.org/10.1177/0735633120972356
65. Vrieze, S.I. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228–243. https://doi.org/10.1037/a0027127
66. Wing, J.M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366 (1881), 3717-3725.
67. Wing, J.M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
68. Voskoglou, M.G. & Buckley, S. (2012). Problem solving and computers in a learning environment. Egyptian Computer Science Journal, 36(4), 28-46.
69. Yağcı, M. (2019). A valid and reliable tool for examining computational thinking skills. Education and Information Technologies, 24(1), 929–951. https://doi.org/10.1007/s10639-018-9801-8