Human and AI Preferences in Turkish Subject Realization: Linguistic and Educational Perspectives

Öz

ABSTRACT: This study investigates whether ChatGPT exhibits human-like sensitivity to the discursive constraints governing the distribution of null and overt subjects in Turkish. The study compares acceptability judgments from 27 native speakers with corresponding evaluations produced by ChatGPT. Participants and the model evaluated 24 contextualized Turkish sentences representing four conditions: Topic Continuity with null (TC-Null) or overt subjects (TC-Overt), and Contrastive Focus with null (CF-Null) or overt subjects (CF-Overt). The results displayed that TC-Null and CF-Overt sentences were judged highly acceptable, while CF-Null sentences received the lowest ratings, and TC-Overt sentences were moderately acceptable by human participants. ChatGPT closely mirrored these patterns, with negligible-to-small effect size differences in three of the four conditions and a small-to-moderate difference only in the CF-Overt condition. These findings suggest that ChatGPT successfully captures distributional regularities at the syntax–discourse interface, despite known limitations in discourse reasoning. The study concludes by discussing implications for linguistic theory and language pedagogy, highlighting the potential role of large language models (LMMs) as supplementary tools for teaching context-sensitive grammatical features in Turkish. Keywords: null and overt subjects, syntax-discourse interface, large language models (LMMs), ChatGPT, Turkish

Anahtar Kelimeler

• null and overt subjects
• syntax-discourse interface
• large language models (LMMs)
• ChatGPT
• Turkish

Türkçede Özne Dağılımında İnsan ve Yapay Zekâ Tercihleri: Dilbilimsel ve Eğitimsel Bakış Açıları

Öz

ÖZ: Bu çalışma, ChatGPT’nin Türkçede boş ve açık öznelerin dağılımını belirleyen söylemsel kısıtlamalara insan benzeri bir duyarlılık gösterip göstermediğini incelemektedir. Çalışmada, 27 anadil Türkçe konuşucusunun kabul edilebilirlik yargıları ile ChatGPT tarafından üretilen değerlendirmeler karşılaştırılmıştır. Katılımcılar ve ChatGPT, 4 farklı koşulu temsil eden toplam 24 bağlamlandırılmış Türkçe tümceyi değerlendirmiştir: boş ya da açık özne içeren konu devamı bağlamları ve boş ve açık özne içeren karşıtsal odak bağlamları. Bulgular, boş özne içeren konu sürekliliği bağlamları ile açık özne içeren karşıtsal odak bağlamlarının yüksek düzeyde kabul edilebilir bulunduğunu; boş özne içeren karşıtsal odak bağlamlarının en düşük puanları aldığını ve açık özne içeren konu sürekliliği bağlamlarının ise anadil konuşucuları tarafından orta düzeyde kabul edilebilir olarak değerlendirildiğini göstermektedir. ChatGPT, bu değerlendirme örüntülerini büyük ölçüde yansıtmış; dört bağlamdan üçünde etki büyüklükleri küçük düzeyde kalırken, yalnızca açık özne içeren karşıtsal odak bağlamlarında küçük ile orta düzey arasında bir fark ortaya çıkmıştır. Bu bulgular, ChatGPT’nin söylemsel alana ilişkin akıl yürütmedeki bilinen sınırlılıklarına rağmen, sözdizim ile söylem arasındaki etkileşimde ortaya çıkan dağılımsal düzenlilikleri başarılı bir biçimde yakalayabildiğini göstermektedir. Çalışma son olarak, dilbilim ve dil öğretimi açısından çıkarımları tartışarak, büyük dil modellerinin Türkçede bağlama duyarlı dilbilgisel özelliklerin öğretiminde tamamlayıcı araçlar olarak değerlendirilebileceğine işaret etmektedir. Anahtar sözcükler: boş ve açık özneler, sözdizim-söylem arayüzü, büyük dil modelleri, ChatGPT, Türkçe

Anahtar Kelimeler

• boş ve açık özneler
• sözdizim-söylem arayüzü
• büyük dil modelleri
• ChatGPT
• Türkçe

Kaynakça

1. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., …
2. Amodei, D. (2020). Advances in Neural Information Processing Systems, 33, 1877–1901. https://doi.org/10.48550/arXiv.2005.14165
3. Chan, C., Cheng, J., Wang, W., Jiang, Y., Fang, T., Liu, X., & Song, Y. (2024). Exploring the potential of ChatGPT on sentence level relations: A focus on temporal, causal, and discourse relations. In Findings of the Association for Computational Linguistics: EACL 2024 (pp. 684–721). https://doi.org/10.48550/arXiv.2304.14827
4. Chomsky, N. (1981). Lectures on government and binding. Foris.
5. Çınar, O. (2024). Revisiting native grammar through L2 theories: Knowledge and processing of null and overt subject pronouns in Turkish. Nesne, 12(33), 331–350. https://doi.org/10.7816/nesne-12-33-03
6. Enç, M. (1986). Topic switching and pronominal subjects in Turkish. In D. I. Slobin & K. Zimmer (Eds.), Studies in Turkish linguistics (pp. 195–208). John Benjamins.
7. Erguvanlı-Taylan, E. (1986). Pronominal versus zero representation of anaphora in Turkish. In D. I. Slobin & K. Zimmer (Eds.), Studies in Turkish linguistics (pp. 209–231). John Benjamins.
8. Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8, 34–48. https://doi.org/10.1162/tacl_a_00298

9. Fan, Y., Jiang, F., Li, P., & Li, H. (2024). Uncovering the potential of ChatGPT for discourse analysis in dialogue: An empirical study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 16998–17010). https://doi.org/10.48550/arXiv.2305.08391
10. Güçlü, R. (2025). Does ChatGPT argue through structure or stance? A metadiscourse analysis of English argumentative essays across topics. In O. Çınar, F. Başbuğ & H. Aydemir (Eds.), Contemporary studies in linguistics I: IMU Linguistics 15th anniversary commemorative volume (pp. 147–169). Artsürem. https://doi.org/10.7816/imuling-15-25-01X007
11. Jiang, F. K., & Hyland, K. (2025). Metadiscursive nouns in academic argument: ChatGPT vs. student practices. Journal of English for Academic Purposes, 75, Article 101514. https://doi.org/10.1016/j.jeap.2025.101514
12. Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274
13. Kucuk, T. (2024). ChatGPT integrated grammar teaching and learning in EFL classes: A study on Tishk International University students in Erbil, Iraq. Arab World English Journal, 100–111.
14. Lozano, C. (2016). Pragmatic principles in anaphora resolution at the syntax–discourse interface: Advanced English learners of Spanish in the CEDEL2 corpus. In M. Alonso Ramos (Ed.), Spanish learner corpus research: Current trends and future perspectives (pp. 235–265). John Benjamins. https://doi.org/10.1075/scl.78.09loz
15. Noh, K., Oh, E., & Song, S. (2024). Testing language models’ syntactic sensitivity to grammatical constraints: A case study of wanna contraction. Frontiers in Communication, 9, Article 1442093. https://doi.org/10.3389/fcomm.2024.1442093
16. OpenAI. (2023). ChatGPT (Mar 14 version) [Large language model]. https://chat.openai.com/
17. Özsoy, A. S. (1987). The null subject parameter and Turkish. In H. E. Boeschoten & L. T. Verhoeven (Eds.), Studies on modern Turkish: Proceedings of the third conference on Turkish linguistics (pp. 82–90). Tilburg University Press.
18. Öztürk, B. (2001). Turkish as a non-pro-drop language. In E. Erguvanlı-Taylan (Ed.), The verb in Turkish (pp. 239–259). John Benjamins.
19. Pack, A., & Maloney, J. (2023). Using generative artificial intelligence for language education research: Insights from using OpenAI’s ChatGPT. TESOL Quarterly, 57(4), 1571–1582. https://doi.org/10.1002/tesq.3253
20. Rothman, J. (2009). Pragmatic deficits with syntactic consequences?: L2 pronominal subjects and the syntax–pragmatics interface. Journal of Pragmatics, 41(5), 951-973. https://doi.org/10.1016/j.pragma.2008.07.007
21. Shin, H., Lee, S., Jang, D., Song, W., Kim, J., Oh, C., Jo, H., Ahn, Y., Oh, S., Chang, H., Kim, S., & Lee, J. (2025). KoBALT: Korean benchmark for advanced linguistic tasks. arXiv:2505.16125. https://doi.org/10.48550/arXiv.2505.16125
22. Sorace, A. (2011). Pinning down the concept of “interface” in bilingualism. Linguistic Approaches to Bilingualism, 1(1), 1–33. https://doi.org/10.1075/lab.1.1.01sor
23. Turan, Ü. D. (1995). Null vs. overt subjects in Turkish discourse: A centering analysis (Doctoral dissertation). University of Pennsylvania.
24. Waldis, A., Perlitz, Y., Choshen, L., Hou, Y., & Gurevych, I. (2024). Holmes: A benchmark to assess the linguistic competence of language models. Transactions of the Association for Computational Linguistics, 12, 1616–1647. https://doi.org/10.1162/tacl_a_00718
25. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S. F., & Bowman, S. R. (2020). BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8, 377–392. https://doi.org/10.1162/tacl_a_00321
26. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv:2302.11382. https://doi.org/10.48550/arXiv.2302.11382
27. Xiao, Y., & Zhi, Y. (2023). An exploratory study of EFL learners’ use of ChatGPT for language learning tasks. Languages, 8(3), Article 212. https://doi.org/10.3390/languages8030212
28. Yaşar, Y. L. (2025). Türkçede özneye yeniden bakış [Yayımlanmamış yüksek lisans tezi]. Dokuz Eylül Üniversitesi.