ChatGPT vs. DeepSeek: A comparative psychometric evaluation of AI tools in generating multiple-choice questions

Abstract

This study examined the psychometric quality of multiple-choice questions generated by two AI tools, ChatGPT and DeepSeek, within the context of an undergraduate Educational Measurement and Evaluation course. Guided by ten learning outcomes (LOs) aligned with Bloom’s Taxonomy, each tool was prompted to generate one five-option multiple-choice item per LO. Following expert review (Kendall’s W = .58); revisions were made, and the finalized test was administered to 120 students. Item analyses revealed no statistically significant differences between the two AI models regarding item difficulty, discrimination, variance, or reliability. A few items -two from ChatGPT and one from DeepSeek- had suboptimal discrimination indices. Tetrachoric correlation analyses of item pairs generated by the two AI tools for the same LO revealed that only one pair showed a non-significant association, whereas all other pairs demonstrated statistically significant and generally moderate correlations. KR-20 and split-half reliability coefficients reflected acceptable internal consistency for a classroom-based assessment, with the DeepSeek-generated half showing a slightly stronger correlation with total scores. Expert feedback indicated that while AI tools generally produced valid stems and correct answers, most revisions focused on improving distractor quality, highlighting the need for human refinement. Generalizability and Decision studies confirmed consistency in expert ratings and recommended a minimum of seven experts for reliable evaluations. In conclusion, both AI tools demonstrated the capacity to generate psychometrically comparable items, highlighting their potential to support educators and test developers in test construction. The study concludes with practical recommendations for effectively incorporating AI into test development workflows.

Keywords

• ChatGPT
• DeepSeek
• Item generation
• Psychometrics
• Generalizability theory

References

1. Alafnan, M.A. (2025). DeepSeek vs. ChatGPT: A comparative evaluation of AI tools in composition, business writing, and communication tasks. Journal of Artificial Intelligence and Technology, 5, 202-210. https://doi.org/10.37965/jait.2025.0740
2. Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy of educational objectives: Complete edition. Longman.
3. Atılgan, H. (2004). Genellenebilirlik kuramı ve çok değişkenlik kaynaklı Rasch modelinin karşılaştırılmasına ilişkin bir araştırma [A reseach on comparisons of generalizability theory and many facets Rasch measurement] [Unpublished doctoral dissertation]. Hacettepe University.
4. Atılgan, H. (2009). Madde ve test istatistikleri [Item and test statistics]. In H. Atılgan, A. Kan, & N. Doğan (Eds.), Eğitimde ölçme ve değerlendirme [Measurement and evaluation in education] (3rd ed., pp. 293-314). Anı Publishing.
5. Baykul, Y. (2000). Eğitimde ve psikolojide ölçme: Klasik test teorisi ve uygulaması [Measurement in education and psychology: Classical test theory and its applications]. ÖSYM Publications.
6. Bloom, B., Englehart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. Longmans, Green.
7. Brennan, R.L. (2001). Generalizability Theory. Springer-Verlag.
8. Clauser, B. (2008). A review of the EDUG software for generalizability analysis. International Journal of Testing, 8(3), 296-301. https://doi.org/10.1080/15305050802262357

9. Clauser, B.E., Margolis, M.J., & Case, S.M. (2006). Testing for licensure and certification in the professions. In R. L. Brennon (Ed.), Educational measurement (4th ed. pp. 701-730). Praeger Publications.
10. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt, Rinehart, and Winston Inc.
11. Daily Sabah. (2023, October 06). Türkiye's student assessment center to use AI to set exam question. https://www.dailysabah.com/turkiye/education/turkiyes student assessment center-to-use-ai-to-set-exam-question
12. Dancey, C.P., & Reidy, J. (2017). Statistics without maths for psychology. Pearson.
13. DeepSeek AI. (2024). DeepSeek v2: Advancing open source large language models. https://www.deepseek.com
14. Diedenhofen, B., & Musch, J. (2015). cocor: A Comprehensive Solution for the Statistical Comparison of Correlations. PLOS ONE, 10(4), Article e0121945. http://dx.doi.org/10.1371/journal.pone.0121945
15. Downing, S.M., & Haladyna, T.M. (2011). Handbook of test development. Lawrence Erlbaum Associates Publishers.
16. Educational Testing Service. (2025a). e rater® scoring engine. ETS. https://www.ets.org/erater.html
17. Educational Testing Service. (2025b). SpeechRater® scoring engine. ETS. https://www.ets.org/speechrater.html
18. Fraenkel, J.R., Wallen, N.E., & Hyun, H.H. (2012). How to design and evaluate research in education. McGraw-Hill.
19. Gao, H., Hashim, H., & Md Yunus, M. (2025). Assessing the reliability and relevance of DeepSeek in EFL writing evaluation: A generalizability theory approach. Language Testing in Asia, 15(33). https://doi.org/10.1186/s40468-025-00369-6
20. Gierl, M.J., & Haladyna, T.M. (2013). Automatic item generation: Theory & practice. Routledge.
21. Gierl, M.J., Lai, H., & Tanygin, V. (2021). Advanced methods in automatic item generation. Routledge.
22. Gierl, M.J., Lai, H., & Turner, S. (2012). Using automatic item generation to create multiple-choice items for assessments in medical education. Medical Education, 46(8), 757-765. https://doi.org/10.1111/j.1365-2923.2012.04289.x
23. Graduate Management Admission Council. (2009). Fairness of automated essay scoring of GMAT AWA. https://www.gmac.com/market intelligence and research/research library/gmat-test-taker-data/research-reports-gmat-related/fairness-of-automated-essay-scoring-of-gmat-awa
24. Graesser, A.C., Conley, M.W., & Olney, A. (2012). Intelligent tutoring systems. In K. R. Harris, S. Graham, T. Urdan, A. G. Bus, S. Major, & H. L. Swanson (Eds.), APA educational psychology handbook, vol. 3. application to learning and teaching (pp. 451-473). American Psychological Association. https://doi.org/10.1037/13275-018
25. Haladyna, T.M., & Rodriguez, M.C. (2013). Developing and validating test items. Routledge.
26. Irwing, P., & Hughes, D.J. (2018). Test development. In P. Irwing, T. Booth, & D. J. Hughes (Eds.), The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (First ed., pp. 3-48), John Wiley & Sons Ltd. https://doi.org/10.1002/9781118489772.ch1
27. Kanık, M. (2024). The use of ChatGPT in assessment. International Journal of Assessment Tools in Education, 11(3), 608-621. https://doi.org/10.21449/ijate.1379647
28. Kendall, M.G., & Smith, B.B. (1939). The problem of m rankings. The Annals of Mathematical Statistics, 10(3), 275-287. http://www.jstor.org/stable/2235668
29. Kıyak, Y.S., & Emekli, E. (2024). ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: A literature review. Postgraduate Medical Journal, 100(1189), 858-865. https://doi.org/10.1093/postmj/qgae065
30. Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310
31. Leslie, T., & Gierl, M.J. (2023). Using automatic item generation to create multiple-choice questions for pharmacy assessment. American Journal of Pharmaceutical Education, 87(10), 1-7. https://doi.org/10.1016/j.ajpe.2023.100081
32. Liu, Z., He, X., Liu, L., Liu, T., & Zhai, X. (2023). Context matters: A strategy to pre-train language model for science education. In N. Wang, G. Rebolledo-Mendez, V. Dimitrova, N. Matsuda, & O.C. Santos (Eds.), Artificial intelligence in education. Springer. https://doi.org/10.1007/978-3-031-36336-8_103
33. Lo, C.K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), Article 410. https://doi.org/10.3390/educsci13040410
34. Makowski, D., Wiernik, B., Patil, I., Lüdecke, D., & Ben-Shachar, M. (2022). correlation: Methods for Correlation Analysis (Version 0.8.3) [R package]. https://CRAN.R project.org/package=correlation
35. Malik, M., Rehan, S., Zimbittas, G., & Manna, S. (2024). Multiple-choice questions reimagined: Exploring the ethical and pedagogical implications of GenAI for higher education. In T. Fujita (Ed.), Proceedings of the British Society for Research into Learning Mathematics (BSRLM), 44(1). https://bsrlm.org.uk/wp-content/uploads/2024/05/BSRLM-CP-44-1-05.pdf
36. Meng, X.-L., Rosenthal, R., & Rubin, D.B. (1992). Comparing correlated correlation coefficients. Psychological Bulletin, 111(1), 172 175. https://doi.org/10.1037/0033 2909.111.1.172
37. Murphy, K.R., & Davidshofer, C.O. (1991). Psychological testing: Principles and applications. Prantice Hall.
38. Ngo, A., Gupta, S., Perrine, O., Reddy, R., Ershadi, S., & Remick, D. (2024). ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions. Academic Pathology, 11(1), 1-5. https://doi.org/10.1016/j.acpath.2023.100099
39. OECD. (2023). PISA 2022 results (Volume I): The state of learning and equity in education. OECD Publishing. https://www.oecd.org/en/publications/pisa 2022 results volume i_53f23881-en/full-report/adaptive-testing-in-pisa-2022_21364c8d.html
40. OpenAI. (2023). GPT-4 technical report. https://openai.com/research/gpt-4
41. ÖSYM (2024, February 13). ÖSYM Başkanı Ersoy: Yapay zekâ ile soru üreteceğiz [ÖSYM President Ersoy: We will generate questions with artificial intelligence]. https://www.osym.gov.tr/TR,29174/osym-baskani-ersoy-yapay-zeka-ile-soru-uretecegiz 13022024.html
42. Özçelik, D.A. (1992). Ölçme ve değerlendirme [Measurement and assessment]. ÖSYM Publication.
43. Özçelik, D.A. (2013). Test hazırlama kılavuzu [Test preparation manual]. Pegem.
44. R Core Team. (2023). R: A language and environment for statistical computing (Version 4.3.2). R Foundation for Statistical Computing. https://www.R-project.org/
45. Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research (Version 2.3.9) [R package]. Northwestern University. https://CRAN.R project.org/package=psych
46. Robitzsch, A. (2024). sirt: Supplementary Item Response Theory Models (Version 4.1.-15). [R package]. https://CRAN.R-project.org/package=sirt
47. Rudner, L., & Schafer, W. (2002) What teachers need to know about assessment. National Education Association.
48. Rycroft-Smith, L., & Macey, D. (2024). Using AI for question generation in mathematics education: What are the advantages and disadvantages? In T. Fujita (Ed.), Proceedings of the British Society for Research into Learning Mathematics (BSRLM), 44(1). https://bsrlm.org.uk/wp-content/uploads/2024/05/BSRLM-CP-44-1-07.pdf
49. The Princeton Review. (2024). Digital SAT security and fairness. https://www.princetonreview.com/college-advice/digital-security-and-fairness
50. Schober, P., Boer, C., & Schwarte, L.A. (2018). Correlation coefficients: Appropriate use and interpretation. Anesthesia & Analgesia, 126(5), 1763-1768.
51. Ratner, B. (2009) The correlation coefficient: Its values range between +1/−1, or do they? Journal of Targeting, Measurement and Analysis for Marketing, 17, 139 142. https://doi.org/10.1057/jt.2009.5
52. Seldon, A., & Abidoye, O. (2018). The fourth education revolution. University of Buckingham Press.
53. Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. SAGE Publications.
54. Shin, D. (2023). A case study on English test item development training for secondary school teachers using AI tools: Focusing on ChatGPT. Language Research, 59(1), 21 42. https://doi.org/10.30961/lr.2023.59.1.21
55. Society, S., & Group, E.W. (2010). Edug user guide. Edumetrics.
56. Tan, Ş. (2014). Öğretimde ölçme ve değerlendirme: KPSS el kitabı [Assessment and evaluation in instruction: KPSS handbook]. Pegem.
57. Thorndike, R.M., & Thorndike-Christ, T. (2014). Measurement and evaluation in psychology and education. Pearson.
58. Turgut, M.F., & Baykul, Y. (2012). Eğitimde ölçme ve değerlendirme [Measurement and evaluation in education]. Pegem.
59. Urhan, S., Gençaslan, O., & Dost, Ş. (2024). An argumentation experience regarding concepts of calculus with ChatGPT. Interactive Learning Environments, 32(10), 7186 7211. https://doi.org/10.1080/10494820.2024.2308093
60. Ünal, D., Erdem, Z.Ç., & Şahin, Z.G. (2025). Will artificial intelligence succeed in passing this test? Creating an achievement test utilizing ChatGPT. Education & Information Technologies, 30, 17263-17287. https://doi.org/10.1007/s10639-025-13461-4
61. Xiaoyu, W. (2024, June 21). AI Scores High in Gaokao Language Tests, Low in Math. China Daily. https://www.chinadaily.com.cn/a/202406/21/WS6674bb00a31095c51c50a0a9.html
62. Wang, J., & Heung, K. (2025). Educational innovation driven by artificial intelligence: The impact of DeepSeek on teachers’ teaching models. Learning & Education, 14(1), 38-42. https://ojs.piscomed.com/index.php/L-E/article/view/4291
63. Wardat, Y., Tashtoush, M.A., Alali, R., & Jarrah, A.M. (2023). ChatGPT: A revolutionary tool for teaching and learning mathematics. Eurasia Journal of Mathematics, Science and Technology Education, 19(7), Article em2286. https://doi.org/10.29333/ejmste/13272
64. Wickham, H., & Bryan, J. (2023). readxl: Read Excel files (Version 1.4.3) [R package]. https://CRAN.R-project.org/package=readxl
65. Wickham, H., François, R., Henry, L., & Müller, K. (2023). dplyr: A grammar of data manipulation (Version 1.1.4) [R package]. https://CRAN.R-project.org/package=dplyr
66. Willse, J.T. (2014). CTT: Classical Test Theory functions (Version 2.3.3) [R package]. https://CRAN.R-project.org/package=CTT
67. Yogesh, A., Telon, G., Lovely, T.F., & Braiton, M. (2025). A comparative study: Evaluating ChatGPT and DeepSeek AI tools in practice. International Journal of Open Information Technologies, 13(5), 67 70. https://cyberleninka.ru/article/n/a comparative study evaluating-chatgpt-and-deepseek-ai-tools-in-practice
68. Zhai, X. (2025). DeepSeek: Transforming the Foundations of Education. Preprints. https://doi.org/10.20944/preprints202503.1776.v1