Artificial intelligence literacy scale: A study of reliability and validity in Turkish university students

Abstract

This study aims to adapt to Turkish the "Scale for the assessment of non-experts: AI literacy" developed by Laupichler et al. (2023a). The scale consists of 31 items with three sub-dimensions: technical understanding, critical thinking, and practical applications. The data required for the validity and reliability study of the scale were collected from 642 undergraduate and graduate students studying in different departments of a state university in the fall semester of the 2023-2024 academic year. First, CFA was applied to the data according to the factor structure in the original scale, but as acceptable fit values could not be obtained because of the analysis, exploratory factor analysis was performed. In the reliability analysis of the factor structure determined by EFA, KMO was calculated as =0.948. It was determined that the scale items were collected in three factors and explained 61.1% of the total variance ("critical thinking" is 25.8%, "technical knowledge" is 25.2%, and "practical applications" explains 10.2% of the total variance). As a result of EFA, it was seen that the sub-dimensions of some of the items in the original scale had changed, and since the factor load values of the three items were very close to each other, they were removed from the scale. Because of CFA, which was conducted to evaluate whether the data supported the hypothesized relationships between the measured variables, Cronbach’s alpha value was found to be 0.90. As a result of the CFA analysis conducted with the 3 sub-dimensions and 28 items in the scale, the Chi-square value (X²=2.85; df=345, N=317, p< .001), which is the fit index of the model, has a good fit and is significant, SRMR = 0.0545and RMSEA = 0.077 values and fit indices and the model has an acceptable fit.

Keywords

• Artificial intelligence literacy scale
• artificial intelligence competencies
• confirmatory factor analysis
• psychometric measurement
• scale adaptation

References

1. Ayyıldız, H. & Cengiz, E. (2006). A conceptual investigation of structural equation model (SEM) on testing marketing models. Süleyman Demirel University İ.İ.B.F. 11(1), 63-84. https://dergipark.org.tr/tr/download/article-file/194860
2. Bora Semiz, B. & Altunışık R. (2016). An evaluation of various attributes of Likert-likert type scales on response styles in marketing research. Bartın Üniversitesi İİBF Dergisi, 7 (14), 577-598.
3. Büyüköztürk, Ş. (2012). Faktör analizi: Temel kavramlar ve ölçek geliştirmede kullanımı. Kuram ve Uygulamalarda Eğitim Yönetimi, (32), 470-483.
4. Büyüköztürk, Ş., Akgün, Ö.E., Özkahveci, Ö. and Demirel, F. (2004). The validity and reliability study of the Turkish version of the Motivated Strategies for Learning Questionnaire. Educational Scienses: Theory&Practice, 4(2), 231-239.
5. Casal-Otero, L., Catala, A., Fernández-Morante, C., Taboada, M., Cebreiro, B., & Barro, S. (2023). AI literacy in K-12: A systematic literature review. International Journal of STEM Education, 10(1), 29. https://doi.org/10.1186/s40594-023-00418-7
6. Child, D. (2006). The essentials of factor analysis. Continuum, London.
7. Coghlan, S., Miller, T., & Paterson, J. (2021). Good proctor or “big brother”? Ethics of online exam supervision technologies. Philosophy & Technology, 34(4), 1581-1606. https://doi.org/10.1007/s13347-021-00476-1
8. Çokluk Ö, Şekercioğlu G, Büyüköztürk Ş. (2016). Sosyal Bilimler İçin Çok Değişkenli İstatistik SPSS ve LISREL Uygulamaları (4. Baskı). Ankara: Pegem Akademi.

9. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied psychology, 78(1), 98‐104.
10. Coşkun, R., Altunışık, R., Yıldırım, E. & Bayraktaroğlu, S. (2010). Sosyal Bilimlerde Araştırma Yöntemleri SPSS Uygulamaları. Sakarya: Sakarya Yayıncılık.
11. Cox, A. M., & Mazumdar, S. (2022). Defining artificial intelligence for librarians. Journal of Librarianship and Information Science, 0(0). https://doi.org/10.1177/09610006221142029
12. Curtis, C., Gillespie, N., & Lockey, S. (2023). AI-deploying organizations are key to addressing ‘perfect storm’of AI risks. AI and Ethics, 3(1), 145-153. https://doi.org/10.1007/s43681-022-00163-7
13. Dai, Y., Chai, C. S., Lin, P. Y., Jong, M. S. Y., Guo, Y., & Qin, J. (2020). Promoting students’ well-being by developing their readiness for the artificial intelligence age. Sustainability, 12(16), 6597. https://doi.org/10.3390/su12166597
14. Faruqe, F., Watkins, R., & Medsker, L. (2021). Competency model approach to AI literacy: Research-based path from initial framework to model. Advances in Artificial Intelligence and Machine Learning; Research, 2(4), 580-587. DOI: 10.54364/aaiml.2022.1140
15. Fornell, C.& Larcker, DF. (1981). Evaluating structural equation models with un observable variables and measurement error. Journal of Marketing Research, 18(1), 39-50.
16. Harrington D.(2009). Confirmatory Factor Analysis. New York: Oxford University Press; p.21-35.
17. Igami, M. (2020). Artificial intelligence as structural estimation: Deep Blue, Bonanza, and AlphaGo. The Econometrics Journal, 23(3), S1-S24. https://doi.org/10.1093/ectj/utaa005
18. Khine, M.S. (2013). Application of structural equation modeling in educational research and practice. Sense Publishers, Rotterdam / Boston / Taipei.
19. Köklü, N. (1995). Tutumların ölçülmesi ve likert tipi ölçeklerde kullanılan seçenekler. Ankara Üniversitesi Eğitim Bilimleri Fakültesi Dergisi, 28(2), 81-93.
20. Larrazabal, A. J., Nieto, N., Peterson, V., Milone, D. H., & Ferrante, E. (2020). Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences, 117(23), 12592-12594.
21. Laupichler, M.C., Aster, A., Haverkamp, N., & Raupach, T. (2023a). Development of the “Scale for the assessment of non-experts’ AI literacy”–An exploratory factor analysis. Computers in Human Behavior Reports, 12, 100338. DOI: 10.1016/j.chbr.2023.100338
22. Laupichler, M.C., Aster, A., Perschewski, J.O., & Schleiss, J. (2023b). Evaluating AI Courses: A Valid and Reliable Instrument for Assessing Artificial-Intelligence Learning through Comparative Self-Assessment. Education Sciences, 13(10), 978. DOI: 10.3390/educsci13100978
23. Nadler, J.T., Weston, R. & Voyles, E. C. (2015). Stuck in the middle: The use and interpretation of mid-points in items on questionnaires. The Journal of General Psychology, 142(2), 71-89. DOI: 10.1080/00221309.2014.994590
24. Ng, D. T. K., Leung, J. K. L., Chu, K. W. S., & Qiao, M. S. (2021). AI literacy: Definition, teaching, evaluation and ethical issues. Proceedings of the Association for Information Science and Technology, 58(1), 504-509.
25. Office for National Statistics, (2023). Uncertainty and how we measure it for our surveys. https://www.ons.gov.uk/methodology/methodologytopicsandstatisticalconcepts/uncertaintyandhowwemeasureit
26. Özdamar K. (2016). Eğitim, sağlık ve davranış bilimlerinde ölçek ve test geliştirme yapısal eşitlik modellemesi IBM SPSS, IBM SPSS AMOS ve MINITAB uygulamalı. Eskişehir: Nisan Kitabevi.
27. Seçer, İ. (2015). Zorbalıkla başa çıkma stratejileri ölçeğinin geliştirilmesi: Geçerlik ve güvenirlik çalışması. Atatürk Üniversitesi Kazım Karabekir Eğitim Fakültesi Dergisi (30), 85-105.
28. Tabachnick, B.G., & Linda S. (2013). Using multivariate statistics, Pearson, Boston.
29. Vupa, Ö., & Gürünlü Alma, Ö. (2008). Doğrusal regresyon çözümlemesinde çoklu bağlantı probleminin sapan değer içeren küçük örneklemlerde bir simülasyon çalışması ile saptanması ve sonuçları. Selçuk Üniversitesi Fen Fakültesi Fen Dergisi, 2(32), 41-51.
30. Wakita, T., Ueshima, N. & Noguchi, H. (2012). Psychological distance between categories in the Likert scale: Comparing different numbers of options. Educational and Psychological, 72(4): 533-546. https://doi.org/10.1177/0013164411431162
31. Wang, B., Rau, P.-L.P., & Yuan, T. (2022). Measuring user competence in using artificial intelligence: validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324–1337. https://doi.org/10.1080/0144929X.2022.2072768
32. Wienrich, C., & Carolus, A. (2021). Development of an instrument to measure conceptualizations and competencies about conversational agents on the example of smart speakers. Frontiers in Computer Science, 70. DOI: 10.3389/fcomp.2021.685277
33. Yaşlıoğlu, MM. (2017). Sosyal bilimlerde faktör analizi ve geçerlilik: Keşfedici ve doğrulayıcı faktör analizlerinin kullanılması. İstanbul Üniversitesi İşletme Fakültesi Dergisi, 46(74), 74-85.
34. Zhao, L., Wu, X., & Luo, H. (2022). Developing AI literacy for primary and middle school teachers in China: Based on a structural equation modeling analysis. Sustainability, 14(21), 14549.