Podcast-Based Learning and Large Language Models: An Experimental Study on English Learning Motivation

Abstract

This paper investigates the effects of podcast-based learning supported by Notebook Large Language Model (LLM) on English as a foreign language (EFL) motivation in a secondary school setting. Using a true experimental pre-post control group design, 70 students of an Anatolian high school in Konya were randomly assigned to one of the experimental conditions (weekly podcast creation using LLM scaffolding) or the control condition (traditional tasks without media production/LLM) which was content-equated. Motivation was assessed at pre-test and post-test by using the Motivation Towards Language Learning Scale (MTLS). Assumptions of normality and homogeneity were met; missing data were minimal and MCAR. Results showed that there was a large within-group gain for the experimental group (t(35)= 5.93, p<.001, d_z= 1.33) and a non-significant change for the control group. At post-test, the experimental group was better than the control group (independent t: t(68)=3.10, p=.004, d=0.98); an ANCOVA controlling for pre-test supported the group effect. Descriptive gains were greater for intrinsic than extrinsic motivation. Findings are consistent with Dornyei's L2 Motivational Self System and the Self-Determination Theory of motivation and suggest that LLM-supported podcast production offers need-supportive audience-facing practice that has a substantial effect on EFL motivation.

Keywords

• Podcast-based learning
• Large language models
• Learner motivation
• EFL
• Experimental design

References

1. Boyd, D. (2010). Social network sites as networked publics: Affordances, dynamics, and implications. In Z. Papacharissi (Ed.), A networked self: Identity, community, and culture on social network sites (pp. 39–58). Routledge.
2. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Houghton Mifflin.
3. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
4. Cohen, L., Manion, L., & Morrison, K. (2018). Research methods in education (8th ed.). Routledge.
5. Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Sage.
6. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
7. Cumming, G. (2014). The new statistics: Why and how. Psychological Science, 25(1), 7–29. https://doi.org/10.1177/0956797613504966
8. Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. Plenum.

9. Dörnyei, Z. (2005). The psychology of the language learner: Individual differences in second language acquisition. Lawrence Erlbaum.
10. Dörnyei, Z. (2009). The L2 motivational self system. In Z. Dörnyei & E. Ushioda (Eds.), Motivation, language identity and the L2 self (pp. 9–42). Multilingual Matters.
11. Gardner, R. C. (1985). Social psychology and second language learning: The role of attitudes and motivation. Edward Arnold.
12. Gee, J. P. (2004). Situated language and learning: A critique of traditional schooling. Routledge.
13. Greenhow, C., & Lewin, C. (2016). Social media and education: Reconceptualizing formal and informal learning. Learning, Media and Technology, 41(1), 6–30. https://doi.org/10.1080/17439884.2015.1064954
14. Gupta, S. K. (2011). Intention-to-treat concept: A review. Perspectives in Clinical Research, 2(3), 109–112. https://doi.org/10.4103/2229-3485.83221
15. Hasan, M. M., & Hoon, T. B. (2013). Podcast applications in language learning: A review of recent studies. English Language Teaching, 6(2), 128–135. https://doi.org/10.5539/elt.v6n2p128
16. Hayes, A. F., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. Communication Methods and Measures, 14(1), 1–24. https://doi.org/10.1080/19312458.2020.1718629
17. Holmes, W., Bialik, M., & Fadel, C. (2019). Artificial intelligence in education: Promises and implications for teaching and learning. Center for Curriculum Redesign.
18. Hsu, C. K. (2020). The effect of podcasting on college students’ English speaking performance and motivation. Computer Assisted Language Learning, 33(6), 688–708. https://doi.org/10.1080/09588221.2019.1590162
19. Ito, M., Baumer, S., Bittanti, M., Boyd, D., Cody, R., Herr-Stephenson, B., ... Tripp, L. (2010). Hanging out, messing around, and geeking out: Kids living and learning with new media. MIT Press.
20. Jenkins, H. (2009). Confronting the challenges of participatory culture: Media education for the 21st century. MIT Press.
21. Karayazgan, S., & Saracaloğlu, A. S. (2021). İngilizce öğrenmeye yönelik motivasyon ölçeği (İÖYMO): Geçerlik ve güvenirlik çalışması. Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi, 41(2), 857-887. https://doi.org/10.17155/gefd.824346
22. Kasneci, E., Sessler, K., Kühnberger, K.-U., Bannert, M., Mengelkamp, C., et al. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Educational Technology & Society, 26(1), 1–19. https://doi.org/10.1016/j.lindif.2023.102274
23. Kim, S., & Reeves, T. C. (2024). Exploring large language models as intelligent feedback tools in second language writing. Computers & Education, 210, 105018. https://doi.org/10.1016/j.compedu.2024.105018
24. Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. Routledge. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science. Frontiers in Psychology, 4, 863. https://doi.org/10.3389/fpsyg.2013.00863
25. Lankshear, C., & Knobel, M. (2011). New literacies: Everyday practices and social learning (3rd ed.). Open University Press.
26. Levene, H. (1960). Robust tests for equality of variances. In I. Olkin (Ed.), Contributions to probability and statistics (pp. 278–292). Stanford University Press.
27. Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83(404), 1198–1202. https://doi.org/10.1080/01621459.1988.10478722
28. McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
29. McLuhan, M. (2003). Understanding media: The extensions of man. Corte Madera, CA: Gingko Press. (Original work published 1964)
30. Noels, K. A., Pelletier, L. G., Clément, R., & Vallerand, R. J. (2000). Why are you learning a second language? Motivational orientations and self-determination theory. Language Learning, 50(1), 57–85. https://doi.org/10.1111/0023-8333.00111
31. O’Bryan, A., & Hegelheimer, V. (2007). Integrating CALL into the classroom: The role of podcasting in an ESL listening strategies course. ReCALL, 19(2), 162–180. https://doi.org/10.1017/S0958344007000523
32. Postman, N. (2000). The humanism of media ecology. Proceedings of the Media Ecology Association, 1, 10–16.
33. Rahimi, M., & Katal, M. (2012). The role of metacognitive listening strategies awareness and podcast-use readiness in using podcasting for learning English as a foreign language. Computers in Human Behavior, 28(4), 1153–1161. https://doi.org/10.1016/j.chb.2012.01.025
34. Reinders, H., & White, C. (2016). 20 years of learner autonomy and technology: How far have we come and where to next? Language Learning & Technology, 20(2), 143–154.
35. Roll, I., & Wylie, R. (2016). Evolution and revolution in artificial intelligence in education. International Journal of Artificial Intelligence in Education, 26(2), 582–599. https://doi.org/10.1007/s40593-016-0110-3
36. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. Wiley.
37. Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68
38. Selwyn, N. (2016). Education and technology: Key issues and debates (2nd ed.). Bloomsbury.
39. Selwyn, N. (2019). Should robots replace teachers? AI and the future of education. Polity Press.
40. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.
41. Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality. Biometrika, 52(3–4), 591–611. https://doi.org/10.1093/biomet/52.3-4.591
42. Siemens, G. (2014). Connectivism: A learning theory for the digital age. International Journal of Instructional Technology and Distance Learning, 2(1), 3–10.
43. Stanley, G. (2013). Language learning with technology: Ideas for integrating technology into the classroom. Cambridge University Press.
44. Stanley, G. (2019). Language learning with technology: Ideas for integrating technology into the classroom. Cambridge University Press.
45. Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009). Multiple imputation for missing data in epidemiological and clinical research. BMJ, 338, b2393. https://doi.org/10.1136/bmj.b2393
46. Thorne, S. L., & Reinhardt, J. (2008). “Bridging activities,” new media literacies, and advanced foreign language proficiency. CALICO Journal, 25(3), 558–572. https://doi.org/10.1558/cj.v25i3.558-572
47. Ushioda, E. (2011). Motivating learners to speak as themselves. In G. Murray, X. Gao, & T. Lamb (Eds.), Identity, motivation and autonomy in language learning (pp. 11–24). Multilingual Matters.
48. Van Breukelen, G. J. P. (2006). ANCOVA versus change from baseline had more power in randomized studies and more bias in nonrandomized studies. Journal of Clinical Epidemiology, 59(9), 920–925. https://doi.org/10.1016/j.jclinepi.2006.02.007
49. Vandergrift, L., & Goh, C. C. M. (2012). Teaching and learning second language listening: Metacognition in action. Routledge.
50. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
51. Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education—Where are the educators? International Journal of Educational Technology in Higher Education, 16, Article 39. https://doi.org/10.1186/s41239-019-0171-0