Chatbot-Supported Implementation on Pre-Service English Language Teachers' Academic Speaking Skills

Abstract

Artificial intelligence (AI) chatbots, particularly with the advancements in generative artificial intelligence (GAI), have gained prominence in language education. In this context, the main purpose of this study, designed with a mixed-methods experimental design, is to examine the effect of the Call Annie chatbot application on the academic speaking skills of pre-service English language teachers, considering the structural limitations of the EFL context. Furthermore, it aims to examine in depth how this effect occurs through qualitative data. The study's data were collected through speaking achievement tests, response papers, and semi-structured interviews, and the validity and reliability of the research were increased by triangulating the data sources. The quantitative findings of the research revealed a statistically significant difference in the midterm and final grades of the experimental group compared to the control group after a 10-week application. The qualitative findings show that pre-service teachers experienced a learning environment free from stress and anxiety, with personalised instant feedback provided by the chatbot implementation. It was concluded that Call Annie had a positive effect on oral communication skills, and pre-service teachers indicated that they would continue to utilize this tool in future speaking activities.

Keywords

• AI Chatbot
• Call Annie
• Academic Speaking Skills
• Pre-service English language teachers

Supporting Institution

None

Project Number

N/A

Ethical Statement

Ethical Statement: Ethical approval for this study was obtained from the Ondokuz Mayıs University Ethics Committee on 28 February 2025 (Approval Decision No: 2025-201). All participants provided informed consent prior to participation. Participation was voluntary, and all data were collected and stored confidentially in anonymized form.

Thanks

The authors thank all participants for their voluntary participation and the editor and anonymous reviewers for their constructive comments.

Hizmet Öncesi İngilizce Öğretmenlerinin Akademik Konuşma Becerilerinde Chatbot Destekli Uygulama

Abstract

Yapay zeka (YZ) sohbet robotları, özellikle üretken yapay zeka (ÜYZ) alanındaki gelişmelerle birlikte, dil eğitiminde ön plana çıkmıştır. Bu bağlamda, karma yöntemlerle tasarlanmış deneysel bir tasarımla gerçekleştirilen bu çalışmanın temel amacı, EFL bağlamının yapısal sınırlamalarını göz önünde bulundurarak, Call Annie sohbet robotu uygulamasının, hizmet öncesi İngilizce öğretmenlerinin akademik konuşma becerileri üzerindeki etkisini incelemektir. Ayrıca, bu etkinin nitel veriler aracılığıyla nasıl gerçekleştiğini derinlemesine incelemek amaçlanmıştır. Çalışmanın verileri, konuşma başarı testleri, yanıt kağıtları ve yarı yapılandırılmış görüşmeler yoluyla toplanmış ve veri kaynakları üçgenleştirilerek araştırmanın geçerliliği ve güvenilirliği artırılmıştır. Araştırmanın nicel bulguları, 10 haftalık uygulamanın ardından deney grubunun ara ve final notlarında kontrol grubuna kıyasla istatistiksel olarak anlamlı bir fark olduğunu ortaya koymuştur. Nitel bulgular, öğretmen adaylarının chatbot uygulaması sayesinde kişiselleştirilmiş anlık geri bildirimlerin olduğu otantik bir iletişim ortamında stres ve konuşma kaygısından uzak bir öğrenme deneyimi yaşadıklarını göstermektedir. Call Annie'nin sözlü iletişim becerileri üzerinde olumlu bir etkisi olduğu sonucuna varılmış ve öğretmen adayları bu aracı gelecekteki konuşma etkinliklerinde de kullanmaya devam edeceklerini belirtmişlerdir.

Keywords

• Yapay Zeka Sohbet Robotu
• Call Annie
• Akademik Konuşma Becerileri
• Hizmet Öncesi İngilizce Öğretmenleri

Supporting Institution

Hiçbir kurumdan destek alınmamıştır

Project Number

N/A

Ethical Statement

Etik Beyan: Bu çalışma için etik onay, 28 Şubat 2025 tarihinde Ondokuz Mayıs Üniversitesi Etik Kurulu'ndan alınmıştır (Onay Kararı No: 2025-201). Tüm katılımcılar, katılım öncesinde bilgilendirilmiş onam vermiştir. Katılım gönüllü olup, tüm veriler anonim olarak toplanmış ve gizli bir şekilde saklanmıştır.

Thanks

Yazarlar, gönüllü katılımları için tüm katılımcılara ve yapıcı yorumları için editöre ve anonim hakemlere teşekkür ederler.

References

1. Aclan, E. M., & Aziz, N. H. A. (2015). Exploring parliamentary debate as a pedagogical tool to develop English communication skills in EFL/ESL classrooms. International Journal of Applied Linguistics and English Literature, 4(2), 1–16. https://doi.org/10.7575/aiac.ijalel.v.4n.2p.1
2. Aliakbari, M., Barzan, P., Allahveysi, S. P., & Bakhtiarvand, M. (2025). AI-powered chatbots for English conversation practice: Impact on speaking proficiency. Technology Assisted Language Education, 3(3), 45–62. https://doi.org/10.22126/tale.2025.12621.1138
3. Azizimajd, H. (2023). Investigating the impacts of voice-based student-chatbot interactions in the classroom on EFL learners' oral fluency and foreign language speaking anxiety. Technology Assisted Language Education, 1(2), 61–83. https://doi.org/10.22126/tale.2023.2732
4. Berns, A., Mota Macias, J. M., Ruiz-Rube, I., & Dodero, J. M. (2018). Exploring the potential of a 360° video application for foreign language learning. In Proceedings of the Sixth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM '18) (pp. 776–780). New York, NY: Association for Computing Machinery. https://doi.org/10.1145/3284179.3284309
5. Chen, J. (2025). Application research on AI chatbots assisting non-English majors in improving oral English. Academic Journal of Management and Social Sciences, 13(2), 90–93. https://doi.org/10.54097/aayrzn06
6. Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Thousand Oaks, CA: SAGE.
7. Davar, N. F., Dewan, M. A. A., & Zhang, X. (2025). AI chatbots in education: Challenges and opportunities. Information, 16(3), 235. https://doi.org/10.3390/info16030235
8. Deng, X., & Yu, Z. (2023). A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability, 15(4), 2940. https://doi.org/10.3390/su15042940

9. Ding, D., & Yusof, A. M. B. (2025). Investigating the role of AI-powered conversation bots in enhancing L2 speaking skills and reducing speaking anxiety: A mixed methods study. Humanities and Social Sciences Communications, 12, Article 1223. https://doi.org/10.1057/s41599-025-05379-7
10. Dizon, G. (2020). Evaluating intelligent personal assistants for L2 listening and speaking development. Language Learning & Technology, 24(1), 16–26. https://doi.org/10125/44705
11. Du, J., & Daniel, B. K. (2024). Transforming language education: A systematic review of AI-powered chatbots for English as a foreign language speaking practice. Computers and Education: Artificial Intelligence, 6, 100230. https://doi.org/10.1016/j.caeai.2024.100230
12. Duong, T., & Suppasetseree, S. (2024). The effects of an artificial intelligence voice chatbot on improving Vietnamese undergraduate students' English-speaking skills. International Journal of Learning, Teaching and Educational Research, 23(3), 293–321. https://doi.org/10.26803/ijlter.23.3.15
13. Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of chatbot and human task partners. Computers in Human Behavior, 75, 461–468. https://doi.org/10.1016/j.chb.2017.05.045
14. Grab, M. Ö. (2025). Integrated AI chatbot practice: A pathway to improved ESL speaking skills. Social Sciences & Humanities Open, 12, 101933. https://doi.org/10.1016/j.ssaho.2025.101933
15. Hakeem, A., Ali, A., Hussain, R., & Khan, B. (2025). AI-powered chatbot for conversational understanding in Roman Urdu. International Journal of Innovations in Science & Technology, 7(6), 127–138.
16. Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning—Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257. https://doi.org/10.1111/jcal.12610
17. Kim, A., & Su, Y. (2024). How implementing an AI chatbot impacts Korean as a foreign language learners' willingness to communicate in Korean. System, 122, 103256. https://doi.org/10.1016/j.system.2024.103256
18. Kim, H.-S., Cha, Y., & Kim, N. Y. (2021). Effects of AI chatbots on EFL students' communication skills. Korean Journal of English Language and Linguistics, 21, 712–734. https://doi.org/10.15738/kjell.21..202108.712
19. Kim, N.-Y. (2019). A study on the use of artificial intelligence chatbots for improving English grammar skills. Journal of Digital Convergence, 17(8), 37–46. https://doi.org/10.14400/JDC.2019.17.8.037
20. Koç, F. Ş. (2024). The development of listening and speaking skills in EFL via an artificially intelligent chatbot application: A quasi-experimental design study (Doctoral dissertation). Middle East Technical University, Ankara, Turkey. Retrieved from https://hdl.handle.net/11511/107856
21. Koç, F. Ş., & Savaş, P. (2025). The use of artificially intelligent chatbots in English language learning: A systematic meta-synthesis study of articles published between 2010 and 2024. ReCALL, 37(1), 4–21. https://doi.org/10.1017/S0958344024000168
22. Liang, K.-H., Davidson, S., Yuan, X., Panditharatne, S., Chen, C.-Y., Shea, R., ... Yu, Z. (2023). ChatBack: Investigating methods of providing grammatical error feedback in a GUI-based language learning chatbot. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) (pp. 83–99). Toronto, Canada: Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.bea-1.7
23. Liu, J., Sihes, A. J. B., & Lu, Y. (2025). How do generative artificial intelligence (AI) tools and large language models (LLMs) influence language learners' critical thinking in EFL education? A systematic review. Smart Learning Environments, 12, Article 43. https://doi.org/10.1186/s40561-025-00406-0
24. Long, M. H. (1981). Input, interaction, and second-language acquisition. Annals of the New York Academy of Sciences, 379(1), 259–278. https://doi.org/10.1111/j.1749-6632.1981.tb42014.x
25. Mendoza, M. D., Hutajulu, O. Y., & Fibriasari, H. (2023). The utilization of artificial intelligence based chatbot in interactive learning media. Journal of Engineering Education Transformations, 37(2), 174–188. https://doi.org/10.16920/jeet/2023/v37i2/23159
26. Moussalli, S., & Cardoso, W. (2020). Intelligent personal assistants: Can they understand and be understood by accented L2 learners? Computer Assisted Language Learning, 33(8), 865–890. https://doi.org/10.1080/09588221.2019.1595664
27. Nguyen, L. A. D., & Le, T. T. P. (2025). Exploring the effects of an AI chatbot on emotional engagement in English speaking lessons: Insights from Call Annie. International Journal of AI in Language Education, 2(2), 79–99. https://doi.org/10.54855/ijaile.25225
28. Rahman, M. K., Ismail, N. A., Hossain, M. A., & Hossen, M. S. (2025). Students' mindset to adopt AI chatbots for effectiveness of online learning in higher education. Future Business Journal, 11, Article 30. https://doi.org/10.1186/s43093-025-00427-8
29. Schmidt, R. (1990). The role of consciousness in second language learning. Applied Linguistics, 11(2), 129–158. https://doi.org/10.1093/applin/11.2.129
30. Sun, Y., Chen, J., & Sundar, S. S. (2024). Chatbot ads with a human touch: A test of anthropomorphism, interactivity, and narrativity. Journal of Business Research, 172, 114403. https://doi.org/10.1016/j.jbusres.2023.114403
31. Swain, M. (2005). The output hypothesis: Theory and research. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 471–483). Mahwah, NJ: Lawrence Erlbaum Associates.
32. Tai, T. Y., & Chen, H. H.-J. (2020). The impact of intelligent personal assistants on adolescent EFL learners' willingness to communicate. Interactive Learning Environments, 28(7), 1485–1502. https://doi.org/10.1080/10494820.2020.1841801
33. Üstünbaş, Ü. (2024). Hey, GPT, can we have a chat?: A case study on EFL learners' AI speaking practice. International Journal of Modern Education Studies, 8(1), 91–107. https://doi.org/10.51383/ijonmes.2024.318
34. Wan, Y., & Moorhouse, B. L. (2025). Using Call Annie as a generative artificial intelligence speaking partner for language learners. RELC Journal, 56(2), 489–498. https://doi.org/10.1177/00336882231224813
35. Wang, X., Wang, X., Park, S., & Yao, Y. (2025). Users' mental models of generative AI chatbot ecosystems (arXiv:2501.19211 [cs.HC]). https://doi.org/10.48550/arXiv.2501.19211
36. Yang, H., Kim, H., Lee, J. H., & Shin, D. (2022). Implementation of an AI chatbot as an English conversation partner in EFL speaking classes. ReCALL, 34(3), 327–343. https://doi.org/10.1017/S0958344022000039
37. Ye, Y., Deng, J., Liang, Q., & Liu, X. (2022). Using a smartphone-based chatbot in EFL learners' oral tasks. International Journal of Mobile and Blended Learning, 14(1). https://doi.org/10.4018/IJMBL.299405