İnşaat Mühendisliği Lisans Eğitiminde ChatGPT’nin Akademik Performansının Değerlendirilmesi: Malzeme Bilimi ve Yapı Malzemesi Sınavları Üzerine Bir Çalışma

Abstract

Büyük dil modellerinin (BDM) ortaya çıkışıyla birlikte yapay zekâ teknolojilerinde önemli gelişmeler meydana gelmiştir. OpenAI tarafından geliştirilen ChatGPT gibi modeller, bilgiye ulaşma, kod yazma, veri analizinde yardımcı olma, çeviri ve çok dilli iletişim, karmaşık sorunları çözme ve içerik oluşturma gibi konularda sergiledikleri yüksek performansla öne çıkmakta; bu yönleriyle pek çok alanda yenilikçi kullanım imkânları sunmaktadır. Bu çalışmanın temel amacı, önde gelen BDM' lerden biri olan ChatGPT'nin, uzmanlık ve teknik bilgi gerektiren inşaat mühendisliği alanındaki lisans düzeyi sınav sorularını yanıtlama performansını kapsamlı bir şekilde değerlendirmektir. Araştırmada, inşaat mühendisliğinin alt disiplinlerinden malzeme bilgisi ve yapı malzemesi dersleri seçilmiş ve farklı zorluk seviyelerinde ve türdeki sınav soruları kullanılmıştır. ChatGPT' nin bu sorulara verdiği yanıtlar; dersi veren akademisyenler tarafından önceden belirlenen soru puanlarına göre değerlendirilmiştir. ChatGPT notları ve öğrencilerin aynı sınavlarda aldığı notlar istatistiksel yöntemler kullanılarak analiz edilmiştir. Öğrenci ve ChatGPT performanslarının karşılaştırılması amacıyla tüm sınav notları kullanılarak z puanı ve yüzdelikleri hesaplanmış ve t testi yapılmıştır. Malzeme Bilgisi dersi kapsamında yapılan değerlendirmelerde, ChatGPT’nin tanım, kavramsal açıklama ve çoktan seçmeli sorulara yönelik yanıtlarında yüksek doğruluk ve teknik yeterlilik sergilediği görülmüştür. T testi sonuçlarına göre malzeme bilgisi dersi kapsamında ChatGPT’nin ara sınav, yarıyıl sonu ve ham not ortalamaları öğrencilerden anlamlı düzeyde yüksek bulunmuştur. Yapı malzemesi dersi kapsamında, ChatGPT’nin çoktan seçmeli ve kavramsal açık uçlu sorulara verdiği yanıtlar genel olarak doğru, açıklayıcı ve tutarlı bulunmuştur. Ancak grafik üretimi ve görsel yorumlama gerektiren sorularda modelin belirgin sınırlılıkları gözlemlenmiştir. T testi sonuçlarına göre yapı malzemesi dersi kapsamında ChatGPT’nin ara sınav, yarıyıl sonu sınavı ve ham not ortalamaları öğrencilerden anlamlı düzeyde yüksek bulunmuştur. Bu çalışma ChatGPT’nin inşaat mühendisliği alanındaki akademik potansiyeli ve sınırlılıklarını ortaya koymaktadır. Ayrıca ChatGPT’ nin inşaat mühendisliği eğitimindeki yeri konusunda gelecekteki çalışmalara ve tartışmalara ışık tutmayı hedeflemektedir.

Keywords

• ChatGPT
• İnşaat mühendisliği
• Mühendislik eğitimi
• Büyük dil modelleri
• Performans değerlendirmesi

Thanks

Bu çalışmanın tamamlanmasına yönelik değerli katkıları ve destekleri için değerli akademisyenlerimize teşekkürlerimizi sunarız.

Assessing the Academic Performance of ChatGPT in Undergraduate Civil Engineering: A Study on Material Science and Building Materials Exam

Abstract

With the emergence of Large Language Models (LLMs), significant advancements have occurred in artificial intelligence technologies. Models such as ChatGPT, developed by OpenAI, stand out for their high performance in areas like accessing information, writing code, assisting in data analysis, translation and multilingual communication, solving complex problems, and creating content; in these respects, they offer innovative applications across many fields. The primary objective of this study is to comprehensively evaluate the performance of ChatGPT, one of the leading LLMs, in answering undergraduate-level exam questions in the field of civil engineering, which requires expertise and technical knowledge. For this research, the courses of "Material Science" and "Building Materials," sub-disciplines of civil engineering, were selected, and exam questions of varying difficulty levels and types were used. The responses provided by ChatGPT to these questions were evaluated by the course lecturers based on pre-determined scoring criteria. The scores of ChatGPT and the students from the same exams were analyzed using statistical methods. To compare their performances, z-scores, percentiles, and t-tests were utilized for all exam scores. In the evaluations for the Material Science course, ChatGPT demonstrated high accuracy and technical proficiency in its responses to questions involving definitions, conceptual explanations, and multiple-choice formats. According to the t-test results, ChatGPT's average scores for the midterm, final, and raw score were significantly higher than those of the students. For the Building Materials course, ChatGPT's answers to multiple-choice and conceptual open-ended questions were generally found to be accurate, explanatory, and coherent. However, significant limitations were observed in questions requiring graph generation and visual interpretation. The t-test results also showed that ChatGPT's average scores for the midterm, final exam, and raw score were significantly higher than the students' for this course. This study reveals the academic potential and limitations of ChatGPT within the field of civil engineering. Furthermore, it aims to shed light on future studies and discussions regarding the place of ChatGPT in civil engineering education.

Keywords

• ChatGPT
• Civil engineering
• Engineering education
• Large language models (LLMs)
• Performance evaluation

Thanks

We would like to express our gratitude to our esteemed academicians for their valuable contributions and support toward the completion of this study

References

1. Ajevski M., Barker K., Gilbert A., Hardie L., Ryan F. ChatGPT and the future of legal education and practice. Law Teacher 2023; 57(3): 352–364. https://doi.org/10.1080/03069400.2023.2207426
2. Aljindan FK., Qurashi AA., Albalawi IAS., Alanazi AMM., Aljuhani HAM., Almutairi FF., Aldamigh OA., Halawani IR., Alarki SMKZ. ChatGPT conquers the Saudi Medical Licensing Exam: Exploring the accuracy of artificial intelligence in medical knowledge assessment and implications for modern medical education. Cureus 2023; 15(9). https://doi.org/10.7759/CUREUS.45043
3. Aluga M. Application of ChatGPT in civil engineering. East African Journal of Engineering 2023; 6(1): 104–112. https://doi.org/10.37284/EAJE.6.1.1272
4. Arabacı MH., Un B., Genç O. Investigation of the use of large language models in the architecture, engineering, and construction industry. 6th International Conference on Scientific and Innovative Studies 2025.
5. Çetin P., Onan A. Büyük dil modellerinin tıp eğitimde soru yanıtlama sistemlerinde kullanımı: Potansiyel, zorluklar ve gelecek yönelimleri. Fen, Matematik ve Bilgisayar Eğitiminde Yenilikler Dergisi 2025.
6. DeVore JL. Probability and statistics for engineering and the sciences. 9th ed. 2017.
7. Doane DP., Seward LE. Measuring skewness: A forgotten statistic? Journal of Statistics Education 2011; 19(2). https://doi.org/10.1080/10691898.2011.11889611
8. Durgun S., Genç O. Evaluating the impact of prompt engineering on ChatGPT’s performance in civil engineering education. 4th International Conference on Trends in Advanced Research 2025.

9. Ellis AR., Slade E. A new era of learning: Considerations for ChatGPT as a tool to enhance statistics and data science education. Journal of Statistics and Data Science Education 2023; 31(2): 128–133. https://doi.org/10.1080/26939169.2023.2223609
10. Genç O. Assessing the role of AI in advancing construction sector industrial symbiosis research: A comparative study of leading digital assistants. Environment, Development and Sustainability 2024; 1–24. https://doi.org/10.1007/S10668-024-05794-W
11. Genç O. Evaluating the effectiveness of AI-powered conversational models in construction management education: A comparative investigation of student performances and ChatGPT. Osmaniye Korkut Ata Üniversitesi Fen Bilimleri Enstitüsü Dergisi 2025a; 8(2): 598–609. https://doi.org/10.47495/OKUFBED.1540360
12. Genç O. Utilizing advanced language models to identify industrial symbiosis opportunities within the circular economy: capabilities and challenges. Engineering, Construction and Architectural Management 2025.
13. Genç O. Is conversational AI ready for engineering licensure? A ChatGPT performance benchmark and prompting strategy evaluation. Journal of Asian Architecture and Building Engineering 2025b. https://doi.org/10.1080/13467581.2025.2574556
14. Genç O. From zero-shot to role-aware reasoning: Exploring prompt strategies for AI-assisted industrial symbiosis identification. 7th International Conference on Innovative Academic Studies 2025c.
15. Hendrycks D., Mazeika M., Kadavath S., Song D. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems 2019; 32. https://arxiv.org/pdf/1906.12340
16. Hope B., Bracey J., Choukir S., Warner D. Assessment of ChatGPT for engineering statics analysis. 2025. http://arxiv.org/abs/2502.00562
17. Hoşer M. Metinden görüntüye yapay zekâ sistemlerinin iç mimari tasarıma katkısının değerlendirilmesi. İstanbul Sabahattin Zaim Üniversitesi Lisansüstü Eğitim Enstitüsü Yüksek Lisans Tezi, İstanbul, 2024.
18. Hotelling H., Solomons LM. The limits of a measure of skewness. Annals of Mathematical Statistics 1932; 3(2): 141–142. https://doi.org/10.1214/AOMS/1177732911
19. Jordan MI., Mitchell TM. Machine learning: Trends, perspectives, and prospects. Science 2015; 349(6245): 255–260. https://doi.org/10.1126/SCIENCE.AAA8415
20. Jurafsky D., Martin JH. Speech and language processing. 3rd ed. 2018.
21. Karakose T., Demirkol M., Aslan N., Köse H., Yirci R. A conversation with ChatGPT about the impact of the COVID-19 pandemic on education: Comparative review based on human–AI collaboration. Educational Process: International Journal 2023; 12(3): 7–25. https://doi.org/10.22521/EDUPIJ.2023.123.1
22. Keiper MC., Fried G., Lupinek J., Nordstrom H. Artificial intelligence in sport management education: Playing the AI game with ChatGPT. Journal of Hospitality, Leisure, Sport and Tourism Education 2023; 33: 100456. https://doi.org/10.1016/J.JHLSTE.2023.100456
23. Kılıç ME., Zmir İ., Corresponding T. AI in medical education: A comparative analysis of GPT-4 and GPT-3.5 on Turkish Medical Specialization Exam performance. MedRxiv 2023. https://doi.org/10.1101/2023.07.12.23292564
24. Koga S. Exploring the pitfalls of large language models: Inconsistency and inaccuracy in answering pathology board examination-style questions. Pathology International 2023; 73(12): 618–620. https://doi.org/10.1111/PIN.13382
25. Kojima T., Gu SS., Reid M., Matsuo Y., Iwasawa Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems 2022; 35. https://arxiv.org/pdf/2205.11916
26. Kung TH., Cheatham M., Medenilla A., Sillos C., De Leon L., Elepaño C., Madriaga M., Aggabao R., Diaz-Candido G., Maningo J., Tseng V. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health 2023; 2(2): e0000198. https://doi.org/10.1371/JOURNAL.PDIG.0000198
27. Lecun Y., Bengio Y., Hinton G. Deep learning. Nature 2015; 521(7553): 436–444. https://doi.org/10.1038/NATURE14539
28. Lee H. Using ChatGPT as a learning tool in acupuncture education: Comparative study. JMIR Medical Education 2023; 9. https://doi.org/10.2196/47427
29. Madaan A., Tandon N., Gupta P., Hallinan S., Gao L., Wiegreffe S., Alon U., Dziri N., Prabhumoye S., Yang Y., Gupta S., Majumder B.P., Hermann K., Welleck S., Yazdanbakhsh A., Clark P. Self-refine: Iterative refinement with self-feedback. NeurIPS 2023.
30. McHugh ML. Interrater reliability: The kappa statistic. Biochemia Medica 2012; 22(3): 276–282. https://doi.org/10.11613/BM.2012.031
31. Nikolic S., Daniel S., Haque R., Belkina M., Hassan GM., Grundy S., Lyden S., Neal P., Sandison C. ChatGPT versus engineering education assessment: A multidisciplinary and multi-institutional benchmarking and analysis. European Journal of Engineering Education 2023; 48(4): 559–614. https://doi.org/10.1080/03043797.2023.2213169
32. Pattier D. ChatGPT and education: A systematic review. Revista Entropía Educativa 2024; 2(3): 80-102.
33. Pavlick E., Kwiatkowski T. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics 2019; 7: 677–694. https://doi.org/10.1162/TACL_A_00293
34. Russell, S., Norvig, P. Artificial intelligence: A modern approach (3rd ed.). Pearson. 2010.
35. Sallam M., Salim NA., Barakat M., Al-Tammemi AB. ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra Journal 2023; 3(1): e103. https://doi.org/10.52225/NARRA.V3I1.103
36. Schulhoff S., Ilie M., Balepur N., Kahadze K., Liu A., Si C., Li Y., Gupta A., Han H., Schulhoff S., Dulepet PS., Vidyadhara S., Ki D., Agrawal S., Pham C., Kroiz G., Li F., Tao H., Srivastava A., Resnik P. The prompt report: A systematic survey of prompt engineering techniques. 2024.
37. Talha Junaid M., Barakat S., Awad R., Anwar N. Adopting the power of AI chatbots for enriching students’ learning in civil engineering education: A study on capabilities and limitations. Studies in Big Data 2024; 144: 25–47. https://doi.org/10.1007/978-3-031-52280-2_3
38. Uddin SMJ., Albert A., Tamanna M., Ovid A., Alsharef A. ChatGPT as an educational resource for civil engineering students. Computer Applications in Engineering Education 2024; 32(4): e22747. https://doi.org/10.1002/CAE.22747
39. Xie M., Meng F., Zou J., Feng W., Ma S. Application of artificial intelligence in civil engineering education. Journal of Physics: Conference Series 2020; 1575(1): 012225. https://doi.org/10.1088/1742-6596/1575/1/012225
40. Xu X., Tao C., Shen T., Xu C., Xu H., Long G., Lou, JG. Re-reading improves reasoning in language models. Conference on Empirical Methods in Natural Language Processing. 2024.
41. Zhui L., Fenghe L., Xuehu W., Qining F., Wei R. Ethical considerations and fundamental principles of large language models in medical education: Viewpoint. Journal of Medical Internet Research 2024; 26(1): e60083. https://doi.org/10.2196/60083