Analysis of The Performance of Artificial Intelligence-Based Language Models in Electrical and Electronics Education Exam Questions

Öz

Over the last decade, advances in artificial intelligence have begun to transform educational practices. With the development of OpenAI's ChatGPT, it has become clear that AI chatbots' capabilities—such as generating human-like text, facilitating automated conversations, and answering questions—will have widespread impact across various sectors, including education and healthcare. Consequently, the use of chatbots in various applications has accelerated. Today's most popular chatbots, OpenAI ChatGPT and large language models (LLMs) like Google Gemini and Bard, are making significant progress with applications in various fields, including electrical and electronics education. In this study, the performance of ChatGPT (GPT-3.5), ChatGPT (GPT-5.1), Bard and Gemini was analyzed in answering 240 carefully prepared questions in Turkish language using clear, specific and plain text formats to determine knowledge levels, covering 12 sub-specialties in the field of electrical and electronics. Two experts analyzed the accuracy metric in the performance analysis, and the results are presented comparatively. The statistical significance of the findings was also determined, and it was observed that there were significant differences for the exam evaluations of some courses.

Anahtar Kelimeler

• Artificial Intelligence
• ChatGPT
• Gemini
• Bard
• Electrical-Electronics Education

Elektrik ve Elektronik Eğitimi Sınav Sorularında Yapay Zeka Temelli Dil Modellerinin Performansının Analizi

Öz

Son on yılda, yapay zekâ alanındaki gelişmeler eğitim uygulamalarını dönüştürmeye başladı. OpenAI'nin ChatGPT'sinin geliştirilmesiyle, yapay zekâ sohbet robotlarının yeteneklerinin (insan benzeri metin üretme, otomatik konuşmaları kolaylaştırma ve soruları yanıtlama gibi) eğitim ve sağlık hizmetleri de dahil olmak üzere çeşitli sektörlerde yaygın bir etkiye sahip olacağı açık hale geldi. Sonuç olarak, çeşitli uygulamalarda sohbet robotlarının kullanımı hızlandı. Günümüzün en popüler sohbet robotları olan OpenAI ChatGPT ve Google Gemini ve Bard gibi büyük dil modelleri (LLM'ler), elektrik ve elektronik eğitimi de dahil olmak üzere çeşitli alanlarda uygulamalarla önemli ilerleme kaydetmektedir. Bu çalışmada, elektrik ve elektronik alanındaki 12 alt uzmanlığı kapsayan, bilgi düzeylerini belirlemek için Türkçe dilinde açık, belirli ve düz metin biçimleri kullanılarak 240 adet dikkatlice hazırlanmış soruyu yanıtlamada ChatGPT (GPT-3.5), ChatGPT (GPT-5.1), Bard ve Gemini'nin performansı analiz edilmiştir. Performans analizinde doğruluk metriği iki uzman tarafından incelenmiş ve sonuçlar karşılaştırmalı olarak sunulmuştur. Bulguların istatistiksel anlamlılığı da belirlenmiş ve bazı derslerin sınav değerlendirmelerinde anlamlı farklılıklar olduğu görülmüştür.

Anahtar Kelimeler

• Yapay Zeka
• ChatGPT
• Gemini
• Bard
• Elektrik-Elektronik Eğitimi

Kaynakça

1. Ali, R., Tang, O. Y., Connolly, I. D., Fridley, J. S., Shin, J. H., Sullivan, P. L. Z., ... , Asaad, W. F., 2023. Performance of ChatGPT, GPT-4, and Google Bard on a neurosurgery oral boards preparation question bank. Neurosurgery, 93(5), 1090-1098.
2. Al-Thani, S. N., Anjum, S., Bhutta, Z. A., Bashir, S., Majeed, M. A., Khan, A. S., Bashir, K., 2025. Comparative performance of ChatGPT, Gemini, and final-year emergency medicine clerkship students in answering multiple-choice questions: implications for the use of AI in medical education. International Journal of Emergency Medicine, 18(1), 146.
3. Baytak, A., 2023. The acceptance and diffusion of generative artificial intelligence in education: A literature review. Current Perspectives in Educational Research, 6(1), 7-18.
4. Calonge, D. S., Smail, L., Kamalov, F., 2023. Enough of the chit-chat: A comparative analysis of four AI chatbots for calculus and statistics. Journal of Applied Learning and Teaching, 6(2), 346-357.
5. Canleblebici, M., Dal, A., Erdag, M., 2024. A comparative analysis to evaluate the effectiveness of Bing, Bard, ChatGPT-3.5 and ChatGPT-4.0 in answering glaucoma-related questions. Turkish Journal of Clinical Experimental Ophthalmology (TJ-CEO), 19(4).
6. Carlà, M. M., Giannuzzi, F., Boselli, F., Rizzo, S., 2024. Testing the power of Google DeepMind: Gemini versus ChatGPT 4 facing a European ophthalmology examination. AJO International, 1(3), 100063.
7. Cheong, R. C. T., Pang, K. P., Unadkat, S., Mcneillis, V., Williamson, A., Joseph, J., ... , Paleri, V., 2024. Performance of artificial intelligence chatbots in sleep medicine certification board exams: ChatGPT versus Google Bard. European Archives of Oto-Rhino-Laryngology, 281(4), 2137-2143.
8. Crawford, J., Cowling, M., Allen, K. A., 2023. Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). Journal of University Teaching and Learning Practice, 20(3), 1-19.

9. Dengel, A., Gehrlein, R., Fernes, D., Görlich, S., Maurer, J., Pham, H. H., ... , Eisermann, N. D. G. 2023. Qualitative research methods for large language models: Conducting semi-structured interviews with ChatGPT and BARD on computer science education. In Informatics (Vol. 10, No. 4, p. 78). MDPI.
10. Dhanvijay, A. K. D., Pinjar, M. J., Dhokane, N., Sorte, S. R., Kumari, A., Mondal, H., 2023. Performance of large language models (ChatGPT, Bing Search, and Google Bard) in solving case vignettes in physiology. Cureus, 15(8).
11. Eldem, A., Eldem, H., 2026. The development and evaluation of agricultural question-answering systems based on large language models. Scientific Reports, 16(1), 5357.
12. Farhat, F., Chaudry, B. M., Nadeem, M., Sohail, S. S., Madsen, D. O., 2023. Evaluating AI models for the National Pre-Medical Exam in India: a head-to-head analysis of ChatGPT-3.5, GPT-4 and Bard. JMIR Medical Education. Preprints.
13. Gan, R. K., Ogbodo, J. C., Wee, Y. Z., Gan, A. Z., González, P. A., 2024. Performance of Google bard and ChatGPT in mass casualty incidents triage. The American journal of emergency medicine, 75, 72-78.
14. Google. Bard. https://bard.google.com (Accessed January 30, 2024)
15. Google. Gemini. https://gemini.google.com (Accessed November 23, 2025)
16. Grassini, S., 2023. Shaping the future of education: Exploring the potential and consequences of AI and ChatGPT in educational settings. Education sciences, 13(7), 692.
17. Hallal, K., Hamdan, R., Tlais, S., 2023. Exploring the potential of AI-Chatbots in organic chemistry: An assessment of ChatGPT and Bard. Computers and Education: Artificial Intelligence, 5, 100170.
18. Hans, F., 2023. ChatGPT vs. Bard-Which is better at solving coding problems. Online Manuscript, 8.
19. Heston, T. F., Khun, C., 2023. Prompt engineering in medical education. International Medical Education, 2(3), 198-205.
20. Hoch, C. C., Wollenberg, B., Lüers, J. C., Knoedler, S., Knoedler, L., Frank, K., ... , Alfertshofer, M., 2023. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. European Archives of Oto-Rhino-Laryngology, 280(9), 4271-4278.
21. Huallpa, J. J., 2023. Exploring the ethical considerations of using Chat GPT in university education. Periodicals of Engineering and Natural Sciences, 11(4), 105-115.
22. Huang, R. S., Lu, K. J. Q., Meaney, C., Kemppainen, J., Punnett, A., Leung, F. H., 2023. Assessment of resident and AI chatbot performance on the University of Toronto family medicine residency progress test: comparative study. JMIR Medical Education, 9, e50514.
23. Jere, S., 2025. Evaluating artificial intelligence large language models’ performances in a South African high school chemistry exam. Eurasia Journal of Mathematics, Science and Technology Education, 21(2), em2582.
24. King, M. R., 2023. Can Bard, Google’s experimental Chatbot based on the LaMDA large language model, help to analyze the gender and racial diversity of authors in your cited scientific references?. Cellular and molecular bioengineering, 16(2), 175-179.
25. Koga, S., 2023. Exploring the pitfalls of large language models: inconsistency and inaccuracy in answering pathology board examination-style questions. medRxiv, 2023-08.
26. Laato, S., Morschheuser, B., Hamari, J., Björne, J., 2023. AI-assisted learning with ChatGPT and large language models: Implications for higher education. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT) (pp. 226-230). IEEE.
27. Lande, D., Strashnoy, L., 2023. Gpt semantic networking: A dream of the semantic web–the time is now.
28. Li, Y., Huang, S., Qi, J., Lu, Y., Quan, L., Han, D., ... , Luan, Z., 2026. Evaluating Large‐Scale and Lightweight Large Language Models for Traditional Chinese Medicine Exam Questions: A Comparative Study. Journal of Evidence‐Based Medicine, e70118.
29. Lim, Z. W., Pushpanathan, K., Yew, S. M. E., Lai, Y., Sun, C. H., Lam, J. S. H., ... , Tham, Y. C., 2023. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine, 95.
30. Luo, D., Liu, M., Yu, R., Liu, Y., Jiang, W., Fan, Q., ... , Zheng, Z., 2025. Evaluating the performance of GPT-3.5, GPT-4, and GPT-4o in the Chinese National Medical Licensing Examination. Scientific reports, 15(1), 14119.
31. Madrid-García, A., Rosales-Rosado, Z., Freites-Nuñez, D., Pérez-Sancristóbal, I., Pato-Cour, E., Plasencia-Rodríguez, C., ... , Rodríguez-Rodríguez, L., 2023. Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training. Scientific Reports, 13(1), 22129.
32. Mahat, R. K., Jantikar, A. M., Rathore, V., Panda, S., 2023. Assessing the performance of ChatGPT to solve biochemistry question papers of university examination. Advances in Physiology Education, 47(3), 528-529.
33. Martlı, E. P., Dincer, N. U., 2021. Technology in nursing education: Augmented reality. Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, 27(5), 627-637.
34. Meo, S. A., Al-Khlaiwi, T., AbuKhalaf, A. A., Meo, A. S., Klonoff, D. C., 2025. The scientific knowledge of Bard and ChatGPT in endocrinology, diabetes, and diabetes technology: multiple-choice questions examination-based performance. Journal of Diabetes Science and Technology, 19(3), 705-710.
35. Nguyen, D., Swanson, D., Newbury, A., Kim, Y. H., 2024. Evaluation of ChatGPT and Google Bard using prompt engineering in cancer screening algorithms. Academic Radiology, 31(5), 1799-1804.
36. Oblitas, L., Adeeb, S., Cruz-Noguez, C., 2024. Performance of ChatGPT and Gemini on the Fundamentals of Engineering (FE) Civil Exam: A Preliminary Comparison and Potential Implications for Civil Engineering Education. Available at SSRN 5038555.
37. Oh, S., Park, J., Baek, H., 2026. Performance Evaluation of OpenAI’s o4-mini on CSAT Mathematics: Multimodal Reasoning and the Reasoning–Verification–Reanalysis Loop. IEEE Access.
38. OpenAI. ChatGPT. https://chat.openai.com (Accessed January 30, 2024)
39. OpenAI. ChatGPT. https://chat.openai.com (Accessed November 23, 2025)
40. Patil, N. S., Huang, R. S., van der Pol, C. B., Larocque, N., 2024. Comparative performance of ChatGPT and Bard in a text-based radiology knowledge assessment. Canadian Association of Radiologists Journal, 75(2), 344-350.
41. Pichai S, Hassabis D., 2023. Introducing Gemini: our largest and most capable AI model. https://blog.google/intl/en-africa/company-news/technology/introducing-gemini-our-largest-and-most-capable-ai-model/
42. Plevris, V., Papazafeiropoulos, G., Jiménez Rios, A., 2023. Dataset of the study: Chatbots put to the test in math and logic problems: A preliminary comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. Zenodo.
43. Pressman, S. M., Borna, S., Gomez-Cabello, C. A., Haider, S. A., Forte, A. J., 2024. AI in hand surgery: assessing large language models in the classification and management of hand injuries. Journal of Clinical Medicine, 13(10), 2832.
44. Rahsepar, A. A., Tavakoli, N., Kim, G. H. J., Hassani, C., Abtin, F., Bedayat, A., 2023. How AI responds to common lung cancer questions: ChatGPT versus Google Bard. Radiology, 307(5), e230922.
45. Ram, B., Verma, P., 2023. Artificial intelligence AI-based chatbot study of ChatGPT, Google AI Bard and Baidu AI. World Journal of Advanced Engineering Technology and Sciences, 8(01), 258-261.
46. Rasul, T., Nair, S., Kalendra, D., Robin, M., de Oliveira Santini, F., Ladeira, W., ... Heathcote, L., 2023. The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied Learning Teaching, 6(1), 41-56.
47. Ray, S. S., Peddinti, P. R., Verma, R. K., Puppala, H., Kim, B., Singh, A., Kwon, Y. N., 2024. Leveraging ChatGPT and Bard: What does it convey for water treatment/desalination and harvesting sectors?. Desalination, 570, 117085.
48. Sandoval, Z. V., 2018. Design and implementation of a chatbot in online higher education settings. Issues in Information Systems, 19(4).
49. Sarker, I. H., 2022. Investigating undergraduate students’ intention to engage in artificial intelligence learning. SN computer science, 3(2), 158-158.
50. Sezgin, E., Chekeni, F., Lee, J., Keim, S., 2023. Clinical accuracy of large language models and Google search responses to postpartum depression questions: cross-sectional study. Journal of Medical Internet Research, 25, e49240.
51. Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., Moy, L., 2023. ChatGPT and other large language models are double-edged swords. Radiology, 307(2), e230163.
52. Tariq, R., Malik, S., Khanna, S., 2024. Evolving landscape of large language models: an evaluation of ChatGPT and Bard in answering patient queries on colonoscopy. Gastroenterology, 166(1), 220-221.
53. Tassoker, M., 2025. ChatGPT-4 Omni’s superiority in answering multiple-choice oral radiology questions. BMC Oral Health, 25(1), 173.
54. Toyama, Y., Harigai, A., Abe, M., Nagano, M., Kawabata, M., Seki, Y., Takase, K., 2024. Performance evaluation of ChatGPT, GPT-4, and Bard on the official board examination of the Japan Radiology Society. Japanese journal of radiology, 42(2), 201-207.
55. Vera, C. L., Picon, I. F., Nunez, M., Gandia, J. A. G., Ancillo, A. D. L., Arroyo, V. R., Figueredo, C. M., 2025. Evaluating large language models on the Spanish Medical Intern Resident (MIR) Examination 2024/2025: A comparative analysis of clinical reasoning and knowledge application. arXiv preprint arXiv:2503.00025.
56. Waisberg, E., Ong, J., Masalkhi, M., Zaman, N., Sarker, P., Lee, A. G., Tavakkoli, A., 2023. A comparative analysis of ChatGPT and Google’s AI’s bard in medicine. New Asian Journal of Medicine, 1(2), 37-43.
57. Watson, S., Romic, J., 2025. ChatGPT and the entangled evolution of society, education, and technology: A systems theory perspective. European Educational Research Journal, 24(2), 205-224.
58. Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., Drachsler, H., 2021. Are we there yet?-a systematic literature review on chatbots in education. Frontiers in artificial intelligence, 4, 654924.
59. Yoseph, E. T., Gonzalez-Suarez, A. D., Lang, S., Desai, A., Hu, S. S., Zygourakis, C. C., 2024. Patient perspectives on AI: a pilot study comparing large language model and physician-generated responses to routine cervical spine surgery questions. Artificial Intelligence Surgery, 4(3), 267-277.