The general attitudes towards artificial intelligence (GAAIS): A meta-analytic reliability generalization study

Öz

This study aims to generalize the reliability of the GAAIS, which is known to perform valid and reliable measurements, is frequently used in the literature, aims to measure one of today's popular topics, and is one of the first examples developed in the field. Within the meta-analytic reliability generalization study, moderator analyses were also conducted on some categorical and continuous variables. Cronbach's α values for the overall scale and the positive and negative subscales, and McDonald's ω coefficients for positive and negative subscales were generalized. Google Scholar, WOS, Taylor & Francis, Science Direct, and EBSCO databases were searched to obtain primary studies. As a result of the screening, 132 studies were found, and these studies were reviewed according to the inclusion criteria. Reliability coefficients obtained from 19 studies that met the criteria were included in the meta-analysis. While meta-analytic reliability generalization was performed according to the random effects model, moderator analyses were performed according to the mixed effect model based on both categorical variables and continuous variables. As a result of the research pooled, Cronbach's α was 0.881, 0.828, and 0.863 for total, the negative, and positive subscales respectively. Also, McDonald's ω was 0.873 and 0.923 for negative and positive subscales respectively. It was found that there were no significant differences between the reliability coefficients for all categorical variables. On the other hand, all continuous moderator variables (mean age, standard deviation age, and rate of female) had a significant effect.

Anahtar Kelimeler

• General attitudes towards artificial intelligence scale
• Reliability generalization
• Cronbach’s α
• Mc Donald’s ω
• Moderator analysis

References

1. Alcocer‐Bruno, C., Ferrer‐Cascales, R., Rubio‐Aparicio, M., & Ruiz‐Robledillo, N. (2020). The medical outcome study‐HIV health survey: A systematic review and reliability generalization meta‐analysis. Research in Nursing & Health, 43(6), 610-620. https://doi.org/10.1002/nur.22070
2. Arslan, K. (2020). Eğitimde yapay zekâ ve uygulamaları [Artificial intelligence and applications in education]. The Western Anatolia Journal of Educational Sciences, 11(1), 71-88. https://dergipark.org.tr/tr/pub/baebd/issue/55426/690058
3. Aslan, Ö.S., Gocen, S., & Sen, S. (2022). Reliability generalization meta-analysis of mathematics anxiety scale for primary school students. Journal of Measurement and Evaluation in Education and Psychology, 13(2), 117 133. https://doi.org/10.21031/epod.1119308
4. Begg, C.B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50(4), 1088. https://doi.org/10.2307/2533446
5. *Bellaiche, L., Shahi, R., Turpin, M.H., Ragnhildstveit, A., Sprockett, S., Barr, N., ... & Seli, P. (2023). Humans versus AI: Whether and why we prefer human-created compared to AI-created artwork. Cognitive Research: Principles and Implications, 8(1), 1-22. https://doi.org/10.1186/s41235-023-00499-6
6. Beretvas, S.N., Meyers, J.L., & Leite, W.L. (2002). A reliability generalization study of the Marlowe Crowne Social Desirability Scale. Educational and Psychological Measurement, 62(4), 570-589. https://doi.org/10.1177/0013164402062004003
7. Beretvas, S.N., Suizzo, M.A., Durham, J.A., & Yarnell, L.M. (2008). A reliability generalization study of scores on Rotter's and Nowicki-Strickland's locus of control scales. Educational and Psychological Measurement, 68(1), 97 119. https://doi.org/10.1177/0013164407301529
8. *Bergdahl, J., Latikka, R., Celuch, M., Savolainen, I., Mantere, E.S., Savela, N., & Oksanen, A. (2023). Self-determination and attitudes toward artificial intelligence: Cross-national and longitudinal perspectives. Telematics and Informatics, 82, 102013. https://doi.org/10.1016/j.tele.2023.102013

9. Borenstein, M., Hedges, L.V., Higgins, J.P., & Rothstein, H.R. (2009). Introduction to meta-analysis. John Wiley & Sons.
10. Breazeal, C. (2004). Designing sociable robots. MIT.
11. Card, N. (2012). Applied meta-analysis for social science research. Guilford.
12. *Carolus, A., Koch, M., Straka, S., Latoschik, M.E., & Wienrich, C. (2023). MAILS-Meta AI Literacy Scale: Development and testing of an AI Literacy Questionnaire based on well-founded competency models and psychological change-and meta-competencies. arXiv preprint. https://doi.org/10.48550/arXiv.2302.09319
13. Caruso, J.C., & Edwards, S. (2001). Reliability generalization of the Junior Eysenck Personality Questionnaire. Personality and Individual Differences, 31, 173 184. https://doi.org/10.1016/S0191-8869(00)00126-4
14. Cochran, W.G. (1954). The combination of estimates from different experiments. Biometrics, 10, 101–129. https://doi.org/10.2307/3001666
15. *Cruz, J.P., Sembekova, A., Omirzakova, D., Bolla, S.R., & Balay-odao, E.M. (2023). General attitudes towards and readiness for medical artificial intelligence among medical and health sciences students in Kazakhstan. https://doi.org/10.2196/preprints.49536.
16. *Darda, K., Carre, M., & Cross, E. (2023). Value attributed to text-based archives generated by artificial intelligence. Royal Society Open Science, 10: 220915. https://doi.org/10.1098/rsos.220915
17. DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled clinical trials, 7(3), 177 188. https://www.biostat.jhsph.edu/~fdominic/teaching/bio656/references/sdarticle.pdf
18. Eser, M.T., & Dogan, N. (2023). Life Satisfaction Scale: A meta-analytic reliability generalization study in Turkey sample. Turkish Psychological Counseling and Guidance Journal, 13(69), 224-239. https://doi.org/10.17066/tpdrd.1223320mn
19. *Gabbiadini, A., Dimitri, O., Cristina, B., & Anna, M. (2023). Does ChatGPT pose a threat to human identity. SSRN, 4377900. https://doi.org/10.2139/ssrn.4377900
20. *Gozzo, M., Woldendorp, M.K., & De Rooij, A. (2021, December). Creative collaboration with the “brain” of a search engine: Effects on cognitive stimulation and evaluation apprehension. In International Conference on ArtsIT, Interactivity and Game Creation (pp. 209-223). Springer International Publishing.
21. Grassini, S. (2023). Development and validation of the AI attitude scale (AIAS-4): A brief measure of general attitude toward artificial intelligence. Frontiers in Psychology, 14: 1191628. https://doi.org/10.3389/fpsyg.2023.1191628
22. *Hadlington, L., Binder, J., Gardner, S., Karanika-Murray, M., & Knight, S. (2023). The use of artificial intelligence in a military context: Development of the Attitudes Toward AI in Defense (AAID) Scale. Frontiers in Psychology, 14, 1164810. https://doi.org/ 10.3389/fpsyg.2023.1164810
23. Hedges, L.V., & Pigott, T.D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9(4), 426 445. https://doi.org/10.1037/1082 989x.9.4.426
24. *Heim, S., & Chan-Olmsted, S. (2023). Consumer trust in AI–human news collaborative continuum: preferences and influencing factors by news production phases. Journalism and Media, 4(3), 946-965. https://doi.org/10.3390/journalmedia4030061
25. Henson, R.K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting “reliability generalization” studies. Measurement and Evaluation in Counseling and Development, 35(2), 113-127. https://doi.org/10.1080/07481756.2002.12069054
26. Hess, T.J., McNab, A.L., & Basoglu, K.S. (2014). Reliability generalization of perceived ease of use, perceived usefulness, and behavioral intentions. MIS Quarterly, 38, 1-28. https://doi.org/10.25300/MISQ/2014/38.1.01
27. Higgins, J.P.T., & Thompson, S.G. (2002), Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539-1558. https://doi.org/10.1002/sim.1186
28. Hopcan, S., Turkmen, G., & Polat, E. (2023). Exploring the artificial intelligence anxiety and machine learning attitudes of teacher candidates. Education and Information Technologies, 1-21. https://doi.org/10.1007/s10639-023-12086-9
29. Huang, S.P. (2018). Effects of using artificial intelligence teaching system for environmental education on environmental knowledge and attitude. Eurasia Journal of Mathematics, Science and Technology Education, 14(7), 3277 3284. https://doi.org/10.29333/ejmste/91248
30. Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016, October). Artificial intelligence and computer science in education: From kindergarten to university. In 2016 Institute of electrical and electronics engineers - Frontiers in education conference (IEEE-FIE) (pp. 1-9). IEEE. https://doi.org/10.1109/FIE.2016.7757570
31. Kaplan, A., & Haenlein, M. (2019). Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Business Horizons, 62(1), 15-25. https://doi.org/10.1016/j.bushor.2018.08.004
32. *Kaya, F., Aydin, F., Schepman, A., Rodway, P., Yetisensoy, O., & Demir Kaya, M. (2022). The roles of personality traits, AI anxiety, and demographic factors in attitudes toward artificial intelligence. International Journal of Human–Computer Interaction, 1-18. https://doi.org/10.1080/10447318.2022.2151730
33. Kieslich, K., Lünich, M., & Marcinkowski, F. (2021). The threats of artificial intelligence scale (TAI) development, measurement and test over three application domains. International Journal of Social Robotics, 13, 1563-1577. https://doi.org/10.1007/s12369-020-00734
34. Kristof, W. (1974). Estimation of reliability and true score variance from a split of a test into three arbitrary parts. Psychometrika, 39, 491-499. https://doi.org/10.1007/BF02291670
35. *Kwak, Y., Ahn, J.W., & Seo, Y.H. (2022). Influence of AI ethics awareness, attitude, anxiety, and self-efficacy on nursing students’ behavioral intentions. BMC Nursing, 21(1), 1-8. https://doi.org/10.1186/s12912-022-01048-0
36. *Kwak, Y., Seo, Y.H., & Ahn, J.W. (2022). Nursing students' intent to use AI-based healthcare technology: Path analysis using the unified theory of acceptance and use of technology. Nurse Education Today, 119: 105541. https://doi.org/10.1016/j.nedt.2022.105541
37. McCarthy, J. (2004). What is artificial intelligence?. http://www.formal.stanford.edu/jmc/whatisai/
38. *Mohamed, H.A., Awad, S.G., Eldiasty, N.E.M.M, & ELsabahy, H.E. (2023). Effect of the artificial intelligence enhancement program on head nurses' managerial competencies and flourishing at work. Egyptian Journal of Health Care, 14(1), 624 645. https://doi.org/10.21608/EJHC.2023.287188
39. Nica, E., Sabie, O.M., Mascu, S., & Luţan, A.G. (2022). Artificial intelligence decision-making in shopping patterns: Consumer values, cognition, and attitudes. Economics, Management and Financial Markets, 17(1), 31 43. https://doi.org/10.22381/emfm17120222.
40. *Nguyen, E. (2023). Trust and algorithmic decision making. UC Santa Barbara, 3(2022), 1-15. https://escholarship.org/content/qt5z86t0dx/qt5z86t0dx.pdf
41. Novick, M.R., & Lewis, C.L. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13. https://doi.org/10.1007/BF02289400
42. Nunnally, J.C. (1978). Psychometric theory (2nd ed.). McGraw-Hill.
43. Osburn, H.G. (2000). Coefficient alpha and related internal consistency reliability coefficients. Psychological Methods, 5(3), 343–355. https://doi.org/10.1037/1082-989X.5.3.343
44. Ozdemir, V., Yildirim, Y., & Tan, S. (2020). A meta-analytic reliability generalization study of the Oxford Happiness Scale in Turkish sample. Journal of Measurement and Evaluation in Education and Psychology, 11(4), 374-404. https://doi.org/10.21031/epod.766266
45. Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron I., Hoffmann, T.C., Mulrow, C.D., …, & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372: 71. https://doi.org/10.1136/bmj.n71
46. Persson, A., Laaksoharju, M., & Koga, H. (2021). We mostly think alike: Individual differences in attitude towards AI in Sweden and Japan. The Review of Socionetwork Strategies, 15(1), 123-142. https://doi.org/10.1007/s12626-021-00071-y
47. Pinto dos Santos, D., Giese, D., Brodehl, S., Chon, S.H., Staab, W., Kleinert, R., ... & Baeßler, B. (2019). Medical students' attitude towards artificial intelligence: A multicentre survey. European radiology, 29, 1640-1646. https://doi.org/10.1007/s00330-018-5601-1
48. Polesie, S., Gillstedt, M., Kittler, H., Lallas, A., Tschandl, P., Zalaudek, I., & Paoli, J. (2020). Attitudes towards artificial intelligence within dermatology: An international online survey. British Journal of Dermatology, 183(1), 159 161. https://doi.org/ 10.1111/bjd.18875
49. Rothstein, H.R., Sutton, A.J., & Borenstein, M. (2005). Publication bias in meta‐analysis: Prevention, assessment and adjustments. John Wiley & Sons.
50. Rosenthal, R. (1979). The ‘‘file drawer problem’’ and tolerance for null results. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638
51. *Saddique, F., Usman, M., Nawaz, M., & Mushtaq, N. (2020). Entrepreneurial orientation and human resource management: The mediating role of Artificial Intelligence. Elementary Education Online, 19(4), 4969-4978. https://doi.org/10.17051/ilkonline.2021.05.777
52. Sánchez-Meca J, Marín-Martínez F, López-López JA, … & López-Nicolás, P. (2021). Improving the reporting quality of reliability generalization meta-analyses: The REGEMA checklist. Research Synthesis Methods, 12, 516 536. https://doi.org/10.1002/jrsm.1487
53. *Schepman, A., & Rodway, P. (2020). Initial validation of the general attitudes towards Artificial Intelligence Scale. Computers in Human Behavior Reports, 1, 100014. https://doi.org/10.1016/j.chbr.2020.100014
54. *Schepman, A., & Rodway, P. (2022). The General Attitudes towards Artificial Intelligence Scale (GAAIS): Confirmatory validation and associations with personality, corporate distrust, and general trust. International Journal of Human–Computer Interaction, 39(13), 2724-2741. https://doi.org/10.1080/10447318.2022.2085400
55. *Seo, Y.H., & Ahn, J.W. (2022). The validity and reliability of the Korean version of the General Attitudes towards Artificial Intelligence Scale for nursing students. The Journal of Korean Academic Society of Nursing Education, 28(4), 357 367. https://doi.org/10.5977/jkasne.2022.28.4.357
56. Sindermann, C., Sha, P., Zhou, M., Wernicke, J., Schmitt, H.S., Li, M., ... & Montag, C. (2021). Assessing the attitude towards artificial intelligence: Introduction of a short measure in German, Chinese, and English language. KI-Künstliche Intelligenz, 35, 109-118. https://doi.org/10.1007/s13218-020-00689-0
57. Thompson, B., & Cook, C. (2002). Stability of the reliability of libqual+™ scores a reliability generalization meta-analysis study. Educational and Psychological Measurement, 62(4), 735-743. https://doi.org/10.1177/0013164402062004013
58. Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174 195. https://doi.org/10.1177/00131640021970448
59. Turkle, S., Breazeal, C., Dasté, O., & Scassellati, B. (2006). Encounters with kismet and cog: Children respond to relational artifacts. Digital media: Transformations in human communication, 120. http://web.mit.edu/people/sturkle/encounterswithkismet.pdf
60. Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58(1), 6–20. https://doi.org/10.1177/0013164498058001002
61. Vacha-Haase, T., Kogan, L.R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60(4), 509 522. https://doi.org/10.1177/00131640021970682
62. Vassar, M.A. (2008). Note on the score reliability for the Satisfaction with Life Scale: An RG study. Soc Indic Res, 86, 47–57. https://doi.org/10.1007/s11205-007-9113-7
63. Waliszewski, K., & Warchlewska, A. (2020). Attitudes towards artificial intelligence in the area of personal financial planning: A case study of selected countries. Entrepreneurship and Sustainability Issues, 8(2), 399-420. https://doi.org/10.9770/jesi.2020.8.2(24)
64. Wallace, K.A., & Wheeler, A.J. (2002). Reliability generalization of the life satisfaction index. Educational and Psychological Measurement, 62(4), 674 684. https://doi.org/10.1177/0013164402062004009
65. *Wang, H., Sun, Q., Gu, L., Lai, K., & He, L. (2022). Diversity in people's reluctance to use medical artificial intelligence: Identifying subgroups through latent profile analysis. Frontiers in Artificial Intelligence, 5: 1006173. https://doi.org/10.3389/frai.2022.1006173
66. Warrens, M.J. (2014). On Cronbach’s alpha as the mean of all possible-split alphas. Advances in Statistics. 742863. https://doi.org/10.1155/2014/742863
67. Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60(2), 201-223. https://doi.org/10.1177/00131640021970466
68. Youngstrom, E.A., & Green, K.W. (2003). Reliability generalization of self-report of emotions when using the Differential Emotions Scale. Educational and Psychological Measurement, 63(2), 279-295. https://doi.org/10.1177/00131644032532
69. Yoruk, S., & Sen, S. (2023). A reliability generalization meta-analysis of the creative achievement questionnaire. Creativity Research Journal, 35(4), 714 729. https://doi.org/10.1080/10400419.2022.2148073
70. Yuzbasioglu, E. (2021). Attitudes and perceptions of dental students towards artificial intelligence. Journal of Dental Education, 85(1), 60 68. https://doi.org/10.1002/jdd.12385