Kimya Sorularının Cevaplanmasında Yapay Zekâ Tabanlı Sohbet Robotlarının Performansının İncelenmesi

Abstract

Yapay zekâ son yıllarda sağlık, bankacılık ve finans, teknoloji, endüstri, psikoloji ve eğitim gibi birçok alanda kullanılmaktadır. Özellikle doğal dili anlayan ve dil modellerini etkili bir şekilde kullanarak cevaplar verebilen yapay zekâ tabanlı sohbet robotlarının (chatbot) ortaya çıkmasıyla beraber sohbet robotlarının sorulara verdikleri cevapların doğruluk düzeyi tartışma konusu olmuştur. Bu araştırmanın amacı, iki sohbet robotunun üniversite seviyesinde, Bloom’un bilişsel alan taksonomisi dikkate alınarak, yüzey gerilimi konusu ile ilgili hazırlanmış sorulara verdikleri cevapların doğruluk düzeylerini belirlemektir. Araştırmanın deseni durum çalışması olarak belirlenmiştir. Veri toplama aracı olarak yüzey gerilimi ile ilgili Bloom’un bilişsel alan taksonomisi dikkate alınarak hazırlanmış altı adet açık uçlu sorudan oluşan ölçek kullanılmıştır. Sohbet robotlarının yüzey gerilimi ile ilgili sorulara verdiği cevaplar üç uzman tarafından değerlendirilmiştir. Araştırmanın bulgularına göre sohbet robotlarının 60 puan üzerinden 35 ve 38 puan aldıkları, aynı sorularda aynı puan ortalamalarına sahip oldukları, çözümleme düzeyindeki soruyu yanlış cevapladıkları, yaratma düzeyindeki sorudan en yüksek puanı aldıkları ve cevaplarında yanlışlıklar/eksiklikler olduğu ancak açıklamalarının %66,7 oranında net olduğu belirlenmiştir. Bu sonuçlardan yola çıkarak; sohbet robotlarının performansının zorluk seviyesi kolaydan zora doğru olan farklı konularda belirlendiği, istem (prompt) girişinin birden fazla yapılarak bu uygulamanın daha doğru cevapların üretilmesine etki edip etmediği ve sohbet robotların cevaplarında yanlış kavramaların olup olmadığının belirlendiği çalışmaların yapılması önerilmektedir.

Keywords

• Yapay zeka
• Sohbet robotu
• Kimya eğitimi
• Bloom'un bilişsel alan taksonomisi

Investigating the Performance of AI-Based Chatbots in Answering Chemistry Questions

Abstract

Artificial intelligence has developed rapidly in recent years and is used in many fields, such as health, banking and finance, technology, industry, psychology and education. Especially with the emergence of artificial intelligence-based chatbots that understand natural language and can answer using language models effectively, the accuracy level of the answers given by chatbots to questions has been a subject of discussion. This study aims to determine the accuracy levels of the answers provided by two chatbots to the questions prepared about surface tension at university level, taking into account Bloom's cognitive domain taxonomy. The research design was determined as a case study. A scale of six open-ended questions about surface tension prepared using Bloom's cognitive domain taxonomy was used as a data collection tool. Three experts evaluated the answers of chatbots to the questions about surface tension. According to the results of the study, the chatbots scored 35 and 38 out of 60 points, they had the same average scores on the same questions, they answered the question at the analysis level incorrectly, they got the highest score on the question at the creation level, and there was misinformation/insufficient information in their answers, but 66.7% of their explanations were clear. Based on these results, it is recommended to carry out studies in which the performance of chatbots is determined in different subjects with difficulty levels from easy to difficult, whether this application affects the production of more accurate answers by making more than one prompt input, and whether there are misconceptions in the responses of chatbots.

Keywords

• Artificial intelligence
• Chatbot
• Chemistry education
• Bloom's cognitive domain taxonomy

References

1. Acar-Sesen, B. & Ince, E. (2010). Internet as a source of misconception. Turkish Online Journal of Educational Technology-TOJET, 9(4), 94-100.
2. AlAfnan, M. A., Dishari, S., Jovic, M. & Lomidze, K. (2023). Chatgpt as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology, 3(2), 60-68. https://doi.org/10.37965/jait.2023.0184
3. Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J. & Wittrock, M. C. (Ed.). (2001). A taxonomy for learning, teaching and assessing. A Revision of Bloom’s Taxonomy of educational objectives. United States: Longman Publishing.
4. Clark, T. M. (2023). Investigating the use of an artificial intelligence chatbot with general chemistry exam questions. Journal of Chemical Education, 100(5), 1905-1916. https://doi.org/10.1021/acs.jchemed.3c00027
5. Das, D., Kumar, N., Longjam, L. A., Sinha, R., Roy, A. D., Mondal, H. & Gupta, P. (2023). Assessing the capability of ChatGPT in answering first- and second-order knowledge questions on microbiology as per Competency-Based Medical Education Curriculum. Cureus, 15(3), e36034. https://doi.org/10.7759/cureus.36034
6. Fergus, S., Botha, M. & Ostovar, M. (2023). Evaluating academic answers generated using ChatGPT. Journal of Chemical Education, 100(4), 1672–1675. https://doi.org/10.1021/acs.jchemed.3c00087
7. Geerling, W., Mateer, G. D., Wooten, J. & Damodaran, N. (2023). Is ChatGPT smarter than a student in principles of economics? SSRN Electronic Journal, 1-24. https://doi.org/10.2139/ssrn.4356034
8. Gilbert, J. K. (2004). Models and modelling: Routes to more authentic science education. International Journal of Science and Mathematics Education, 2, 115–130.

9. Gilbert, J. K. & Watts, D. M. (1983). Concepts, misconceptions and alternative conceptions: Changing perspectives in science education. Studies in Science Education, 10(1), 61-98. https://doi.org/10.1080/03057268308559905
10. Greca, I. M. & Moreira, M. A. (2000). Mental models, conceptual models, and modelling. International Journal of Science Education, 22(1), 1–11.
11. Gregorcic, B. & Pendrill, A. M. (2023). ChatGPT and the frustrated Socrates. Physics Education, 58(3), 035021. https://doi.org/10.1088/1361-6552/acc299
12. Gungordu, N., Yalcin-Celik, A. & Kilic, Z. (2017). Students' misconceptions about the ozone layer and the effect of internet-based media on it. International Electronic Journal of Environmental Education, 7(1), 1-16.
13. Haenlein, M. & Kaplan, A. (2019). A brief history of artificial intelligence: On the past, present, and future of artificial intelligence. California Management Review, 61(4), 5-14. https://doi.org/10.1177/0008125619864
14. Han, Z., Battaglia, F., Udaiyar, A., Fooks, A. & Terlecky, S. R. (2023). An explorative assessment of ChatGPT as an aid in medical education: Use it with caution. https://www.medrxiv.org/content/10.1101/2023.02.13.23285879v1.full.pdf sayfasından erişilmiştir.
15. Humphry, T. & Fuller, A. L. (2023). Potential ChatGPT use in undergraduate chemistry laboratories. Journal of Chemical Education, 100(4), 1434–1436.
16. Hussain, S., Ameri-Sianaki, O. & Ababneh, N. (2019). A survey on conversational agents/chatbots classification and design techniques. Web, Artificial Intelligence and Network Applications: Proceedings of the Workshops of the 33rd International Conference on Advanced Information Networking and Applications (WAINA-2019) 33 içinde (s. 946-956). New York: Springer International Publishing.
17. Jalil, S., Rafi, S., LaToza, T. D., Moran, K. & Lam, W. (2023). ChatGPT and software testing education: Promises & perils. 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) içinde (s. 4130-4137). New York: IEEE.
18. Koo, T. K. & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163.
19. Korsakova, E., Sokolovskaya, O., Minakova, D., Gavronskaya, Y., Maksimenko, N. & Kurushkin, M. (2022). Chemist bot as a helpful personal online training tool for the final chemistry examination. Journal of Chemical Education, 99(2), 1110-1117. https://doi.org/10.1021/acs.jchemed.1c00789
20. Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., ... & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2(2), e0000198. https://doi.org/10.1371/journal.pdig.0000198
21. Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410-455. https://doi.org/10.3390/educsci13040410
22. Meço, G. & Coştu, F. (2002). Eğitimde yapay zekânın kullanılması: Betimsel içerik analizi çalışması. Karadeniz Teknik Üniversitesi Sosyal Bilimler Enstitüsü Sosyal Bilimler Dergisi, 12(23), 171-193.
23. Mhlanga, D. (2023). Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. https://www.researchgate.net/profile/David-Mhlanga-2/publication/ 368476294_Open_AI_in_Education_the_Responsible_and_Ethical_Use_of_ChatGPT_Towards_Lifelong_Learning/links/63eb1c91bd7860764366f597/Open-AI-in-Education-the-Responsible-and-Ethical-Use-of-ChatGPT-Towards-Lifelong-Learning.pdf sayfasından erişilmiştir.
24. Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis (2. b.). California: SAGE.
25. Motlagh, N. Y., Khajavi, M., Sharifi, A. & Ahmadi, M. (2023). The impact of artificial intelligence on the evolution of digital education: A comparative study of openAI text generation tools including ChatGPT, Bing Chat, Bard, and Ernie. https://www.researchgate.net/profile/Negin-Yazdani-Motlagh/publication/373800938_The_Impact_of_Artificial_Intelligence_on_the_ Evolution_of_Digital_Education_A_Comparative_Study_of_OpenAI_Text_Generation_Tools_including_ChatGPT_Bing_Chat_Bard_and_Ernie/links/64fd0144d6fa5c5bc46cfdbd/The-Impact-of-Artificial-Intelligence-on-the-Evolution-of-Digital-Education-A-Comparative-Study-of-OpenAI-Text-Generation-Tools-including-ChatGPT-Bing-Chat-Bard-and-Ernie.pdf sayfasından erişilmiştir.
26. Nakhleh, M. B. (1992). Why some students don't learn chemistry: Chemical misconceptions. Journal of Chemical Education, 69(3), 191-195.
27. Rahaman, M. S., Ahsan, M. M., Anjum, N., Rahman, M. M. & Rahman, M. N. (2023). The AI race is on! Google's Bard and OpenAI's ChatGPT head to head. http://dx.doi.org/10.2139/ssrn.4351785
28. Rudolph, J., Tan, S. & Tan, S. (2023). War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. Journal of Applied Learning and Teaching, 6(1), 364-389. https://doi.org/10.37074/jalt.2023.6.1.23
29. Shawar, B. A. & Atwell, E. (2007). Chatbots: Are they really useful? Journal for Language Technology and Computational Linguistics, 22(1), 29–49.
30. Steenbergen-Hu, S. & Cooper, H. A. (2014). Meta-analysis of the effectiveness of intelligent tutoring systems on college students’ academic learning. Journal of Educational Psychology, 106(2), 331–347.
31. Susnjak, T. (2022). ChatGPT: The end of online exam integrity? Cornell University arXiv. https://doi.org/10.48550/arXiv.2212.09292
32. Suta, P., Lan, X., Wu, B., Mongkolnam, P. & Chan, J. H. (2020). An overview of machine learning in chatbots. International Journal of Mechanical Engineering and Robotics Research, 9(4), 502-510. https://doi.org/10.18178/ijmerr.9.4.502-510
33. Taber, K. (2002). Chemical misconceptions: prevention, diagnosis and cure (c. 1). Londra: Royal Society of Chemistry.
34. Taber, K. (2009). Challenging misconceptions in the chemistry classroom: resources to support teachers. Educació Química(4), 13-20.
35. Tamer, H. Y. & Övgün, B. (2020). Yapay zekâ bağlamında dijital dönüşüm ofisi. Ankara Üniversitesi SBF Dergisi, 75(2), 775-803. https://doi.org/10.33630/ausbf.691119
36. Wood, D. A., Achhpilia, M. P., Adams, M. T., Aghazadeh, S., Akinyele, K., Akpan, M., ... & Kuruppu, C. (2023). The ChatGPT artificial intelligence chatbot: How well does it answer accounting assessment questions? Issues in Accounting Education, 38(4), 81-108. https://doi.org/10.2308/ISSUES-2023-013
37. Yıldırım, A. & Şimşek, H. (2018). Sosyal bilimlerde nitel araştırma yöntemleri (11. b.). Ankara: Seçkin Yayıncılık.
38. Zhu, J. J., Jiang, J., Yang, M. & Ren, Z. J. (2023). ChatGPT and environmental research. Environmental Science & Technology [Special Issue], A-D. https://doi.org/10.1021/acs.est.3c01818