The Use of Artificial Intelligence Tools in Assessing Content Validity: A Comparative Study with Human Experts

Abstract

This study investigated whether artificial intelligence (AI) systems could evaluate the content validity of B1-level English reading comprehension items in a manner comparable to human experts. A 25-item multiple-choice test was developed and rated by four human experts and four AI models (GPT-4o, Gemini 2.0, Replika, and Chatfuel). To ensure methodological transparency, a standardized few-shot prompting framework was employed. Each model was accessed in a new session to avoid context drift, and identical prompts were provided to maintain consistency across evaluations. The Content Validity Ratio (CVR) and Item Content Validity Index (I-CVI) were calculated and compared using the Wilcoxon Signed-Rank Test. The results revealed no statistically significant difference between AI- and human-generated scores, suggesting that AI evaluators produced ratings similar to those of human experts. Although fine-tuning could not be performed due to the closed-source nature of the models, standardized prompting procedures enhanced reliability and replicability. These findings indicate that AI tools can complement expert judgment in content validity assessment when methodological control is ensured. The study highlights the potential of hybrid human–AI frameworks in educational measurement, while emphasizing the importance of model transparency and context management.

Keywords

• content validity
• artificial intelligence
• few-shot prompting
• context drift
• hybrid evaluation
• educational measurement

Kapsam Geçerliği Değerlendirmesinde Yapay Zeka Araçlarının Kullanımı: İnsan Uzmanlarla Karşılaştırmalı Bir Çalışma

Abstract

Bu çalışma, yapay zekâ (YZ) sistemlerinin, B1 düzeyindeki İngilizce okuduğunu anlama maddelerinin kapsam geçerliğini insan uzmanlara benzer bir biçimde değerlendirip değerlendiremeyeceğini araştırmıştır. Bu amaçla, 25 maddeden oluşan çoktan seçmeli bir test geliştirilmiş ve dört insan uzman ile dört YZ modeli (GPT-4o, Gemini 2.0, Replika ve Chatfuel) tarafından değerlendirilmiştir. Yöntemsel şeffaflığı sağlamak için standartlaştırılmış bir few-shot prompting çerçevesi kullanılmıştır. Her model, bağlam kaymasını (context drift) önlemek amacıyla yeni bir oturumda çalıştırılmış ve tüm değerlendirmelerde tutarlılığı korumak için aynı yönergeler verilmiştir. İçerik Geçerliği Oranı (CVR) ve Madde Düzeyinde İçerik Geçerliği İndeksi (I-CVI) hesaplanmış ve Wilcoxon İşaretli Sıra Testi kullanılarak karşılaştırılmıştır. Sonuçlar, YZ ve insan tarafından üretilen puanlar arasında istatistiksel olarak anlamlı bir fark bulunmadığını göstermiştir; bu da YZ değerlendiricilerinin insan uzmanlara benzer puanlamalar yaptığı anlamına gelmektedir. Modellerin kapalı kaynak yapısı nedeniyle fine-tuning (yeniden uyarlama) gerçekleştirilememiş olsa da, standartlaştırılmış yönlendirme prosedürleri güvenilirliği ve yinelenebilirliği artırmıştır. Bulgular, yöntemsel denetim sağlandığında YZ araçlarının kapsam geçerliği değerlendirmesinde uzman yargısını tamamlayıcı bir unsur olarak kullanılabileceğini göstermektedir. Çalışma, eğitimde ölçme alanında insan-YZ iş birliğine dayalı hibrit modellerin potansiyelini vurgularken, model şeffaflığı ve bağlam yönetiminin önemine dikkat çekmektedir.

Keywords

• kapsam geçerliği
• yapay zekâ
• few-shot prompting
• bağlam kayması
• hibrit değerlendirme
• eğitimde ölçme

References

1. Ahmed, H. K., & Hussein, J. A. (2020). Design and implementation of a chatbot for Kurdish language speakers using Chatfuel platform. Kurdistan Journal of Applied Research, 5(2), 117–135. https://doi.org/10.24017/science.2020.2.10
2. Almanasreh, E., Moles, R., & Chen, T. F. (2019). Evaluation of methods used for estimating content validity. Research in Social and Administrative Pharmacy, 15(2), 214–221. https://doi.org/10.1016/j.sapharm.2018.03.066
3. Arqub, S. A., Al-Moghrabi, D., Allareddy, V., Upadhyay, M., Vaid, N., & Yadav, S. (2024). Content analysis of AI-generated (ChatGPT) responses concerning orthodontic clear aligners. The Angle Orthodontist, 94(3), 263–272. https://doi.org/10.2319/071123-484.1
4. Ayre, C., & Scally, A. J. (2014). Critical values for Lawshe’s content validity ratio: Revisiting the original methods of calculation. Measurement and Evaluation in Counseling and Development, 47(1), 79–86. https://doi.org/10.1177/0748175613513808
5. Cohen, R. J., Swerdlik, M. E., & Phillips, S. M. (2021). Psychological testing and assessment. McGraw Hill.
6. Davis, L. L. (1992). Instrument review: Getting the most from a panel of experts. Applied Nursing Research, 5(4), 194–197. https://doi.org/10.1016/S0897-1897(05)80008-4
7. Hassabis, D., & Kavukcuoglu, K. (2024, December 11). Introducing Gemini 2.0: Our new AI model for the agentic era. Google. https://blog.google/innovation-and-ai/models-and-research/google-deepmind/google-gemini-ai-update-december-2024/
8. Fjelland, R. (2020). Why general artificial intelligence will not be realized. Humanities and Social Sciences Communications, 7(1), 1–9. https://doi.org/10.1057/s41599-020-0494-4

9. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
10. Gamer, M., Lemon, J., Fellows, I., & Singh, P. (2019). irr: Various coefficients of interrater reliability and agreement (R package version 0.84.1). https://doi.org/10.32614/CRAN.package.irr
11. Gürdil, H., Soğuksu, Y. B., Salihoğlu, S., & Coşkun, F. (2025). Eğitimde ölçmede yapay zekânın entegrasyonu: Madde tepki kuramı kapsamında veri üretiminde ChatGPT’nin etkililiği. Trakya Eğitim Dergisi, 15(2), 887–918. https://doi.org/10.24315/tred.1509299
12. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023
13. Hayawi, K., Shahriar, S., & Mathew, S. S. (2024). The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD. Journal of Information Science. Advance online publication. https://doi.org/10.1177/01655515241227531
14. Haynes, S. N., Richard, D., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7(3), 238–247. https://doi.org/10.1037/1040-3590.7.3.238
15. Hinkin, T. R., & Tracey, J. B. (1999). An analysis-of-variance approach to content validation. Organizational Research Methods, 2(2), 175–186. https://doi.org/10.1177/109442819922004
16. Kim, H., Baghestani, S., Yin, S., Karatay, Y., Kurt, S., Beck, J., & Karatay, L. (2024). ChatGPT for writing evaluation: Examining the accuracy and reliability of AI-generated scores compared to human raters. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 73–95). Iowa State University Digital Press. https://doi.org/10.31274/isudp.2024.154.06
17. Kim, J., McFee, M., Fang, Q., Abdin, O., & Kim, P. M. (2023). Computational and artificial intelligence-based methods for antibody development. Trends in Pharmacological Sciences, 44(3), 175–189. https://doi.org/10.1016/j.tips.2022.12.005
18. Kizilcec, R. F., Huber, E., Papanastasiou, E. C., Cram, A., Makridis, C. A., Smolansky, A., Zeivots, S., & Raduescu, C. (2024). Perceived impact of generative AI on assessments: Comparing educator and student perspectives in Australia, Cyprus, and the United States. Computers and Education: Artificial Intelligence, 7, 100269. https://doi.org/10.1016/j.caeai.2024.100269
19. Kohnke, L., Moorhouse, B. L., & Zou, D. (2023). ChatGPT for language teaching and learning. RELC Journal, 54(2), 537–550. https://doi.org/10.1177/00336882231162868
20. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
21. Lee, S., & Park, G. (2024). Development and validation of ChatGPT literacy scale. Current Psychology, 43(21), 18992–19004. https://doi.org/10.1007/s12144-024-05723-0
22. Lin, X. (2024). Exploring the role of ChatGPT as a facilitator for motivating self-directed learning among adult learners. Adult Learning, 35(3), 156–166. https://doi.org/10.1177/10451595231184928
23. Low, D. M., Rankin, O., Coppersmith, D. D., Bentley, K. H., Nock, M. K., & Ghosh, S. S. (2024). Using generative AI to create lexicons for interpretable text models with high content validity [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/vf2bc
24. Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386. https://doi.org/10.1097/00006199-198611000-00017
25. MacFarland, T. W., & Yates, J. M. (2016). Kruskal–Wallis H-test for oneway analysis of variance (ANOVA) by ranks. In Introduction to nonparametric statistics for the biological sciences using R (pp. 177–211). Springer International Publishing. https://doi.org/10.1007/978-3-319-30634-6_6
26. Mertler, C. A., Vannatta, R. A., & LaVenia, K. N. (2025). Advanced and multivariate statistical methods: Practical application and interpretation. Routledge. https://doi.org/10.4324/9781003562436
27. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education/Macmillan.
28. Morjaria, L., Burns, L., Bracken, K., Ngo, Q. N., Lee, M., Levinson, A. J., Smith, J., Thompson, P., & Sibbald, M. (2023). Examining the threat of ChatGPT to the validity of short answer assessments in an undergraduate medical program. Journal of Medical Education and Curricular Development, 10, 23821205231204178. https://doi.org/10.1177/23821205231204178
29. Nemt-Allah, M., Khalifa, W., Badawy, M., Elbably, Y., & Ibrahim, A. (2024). Validating the ChatGPT usage scale: Psychometric properties and factor structures among postgraduate students. BMC Psychology, 12, 497. https://doi.org/10.1186/s40359-024-01983-4
30. Newman, I., Lim, J., & Pineda, F. (2013). Content validity using a mixed methods approach: Its application and development through the use of a table of specifications methodology. Journal of Mixed Methods Research, 7(3), 243–260. https://doi.org/10.1177/1558689813476922
31. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
32. Ofgang, E. (2024, May 14). GPT-4o: What educators need to know. Tech & Learning. https://www.techlearning.com/how-to/gpt-4o-how-to-use-it-to-teach
33. Onal, S., & Kulavuz-Onal, D. (2024). A cross-disciplinary examination of the instructional uses of ChatGPT in higher education. Journal of Educational Technology Systems, 52(3), 301–324. https://doi.org/10.1177/00472395231196532
34. OpenAI. (2023). GPT-4 technical report [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2303.08774
35. Pentina, I., Hancock, T., & Xie, T. (2023). Exploring relationship development with social chatbots: A mixed-method study of Replika. Computers in Human Behavior, 140, 107600. https://doi.org/10.1016/j.chb.2022.107600
36. Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
37. R Core Team. (2024). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
38. Saraswati, G. P. D., Farida, A. N., & Yuliati, Y. (2023). Implementing AI Replika in higher education speaking classes: Benefits and challenges. ELT Forum: Journal of English Language Teaching, 12(3), 207–215. https://doi.org/10.15294/elt.v12i3.76525
39. Shane, J. (2019). You look like a thing and I love you. Hachette UK.
40. Siegle, D. (2023). A role for ChatGPT and AI in gifted education. Gifted Child Today, 46(3), 211–219. https://doi.org/10.1177/10762175231168443
41. Sihite, M. R., Meisuri, M., & Sibarani, B. (2023). Examining the validity and reliability of ChatGPT 3.5-generated reading comprehension questions for academic texts. Randwick International of Education and Linguistics Science Journal, 4(4), 937–944. https://doi.org/10.47175/rielsj.v4i4.835
42. Tenhundfeld, N. L. (2023). Two birds with one stone: Writing a paper entitled “ChatGPT as a tool for studying human–AI interaction in the wild” with ChatGPT. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 67(1), 2007–2012. https://doi.org/10.1177/21695067231192916
43. ul haq Akhoon, I., Khan, M. Y., & Bhat, T. A. (2024). Comparative analysis of AI-generated research content: Evaluating ChatGPT and Google Gemini [Preprint]. Research Square. https://doi.org/10.21203/rs.3.rs-5265799/v1
44. Wachinger, J., Bärnighausen, K., Schäfer, L. N., Scott, K., & McMahon, S. A. (2025). Prompts, pearls, imperfections: Comparing ChatGPT and a human researcher in qualitative data analysis. Qualitative Health Research, 35(9), 951–966. https://doi.org/10.1177/10497323241244669
45. Weng, X., Xia, Q., Gu, M., Rajaram, K., & Chiu, T. K. F. (2024). Assessment and learning outcomes for generative AI in higher education: A scoping review on current research status and trends. Australasian Journal of Educational Technology, 40(6), 37–55. https://doi.org/10.14742/ajet.9540
46. Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25(5), 508–518. https://doi.org/10.1177/0193945903252998
47. Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024). A scoping review on how generative artificial intelligence transforms assessment in higher education. International Journal of Educational Technology in Higher Education, 21, 40. https://doi.org/10.1186/s41239-024-00468-z
48. Zacharis, G., & Papadakis, S. (2025). Can AI grade like a human? Validity, reliability, and fairness in university coursework assessment. Educational Process: International Journal, 19, e2025591. https://doi.org/10.22521/edupij.2025.19.591