Türkçe kısa öykülerde yazar tanıma: dilbilimsel ve hesaplamalı özellik setlerinin karşılaştırmalı analizi

Öz

Arka Plan—Yazar tanıma (authorship attribution) bir metnin üslup özelliklerine dayanarak yazarını belirleme sürecidir. İngilizce gibi eklemeli (bitişken) olmayan diller için geliştirilen yöntemler olgunlaşmış olsa da Türkçe gibi sondan eklemeli diller yeterince araştırılmamıştır. Türkçenin bu yapısı İngilizce merkezli yöntemlerin kapsamadığı farklı bir özellik uzayı oluşturmaktadır.

Amaç—Bu çalışma Türkçe kısa öykülerde yazar tanıma amacıyla altı farklı özellik setini beş sınıflandırıcı model üzerinde sistematik olarak karşılaştırmaktadır. Bu özellik setleri stilometrik, morfolojik (Stanza UD ve Zeyrek), işlev sözcükleri, karakter n-gramları (TF-IDF) ve BERTurk cümle vektörleridir (sentence embeddings). Özellik setleri arasındaki boyut dengesizliğinin ablasyon sonuçlarını çarpıtmasını önlemek için PCA ile boyut-eşitlemeli bir ablasyon çerçevesi önerilmektedir.

Yöntem—On yazardan 11’er kısa öykü ile dengeli bir derlem oluşturulmuştur (110 öykü, 135431 sözcük). Altı özellik seti çıkarılmış ve beş sınıflandırıcı (SVM-Doğrusal, SVM-RBF, Lojistik Regresyon, Rastgele Orman, Çok Katmanlı Algılayıcı) 5 katlı tabakalı çapraz doğrulama ile değerlendirilmiştir. Standart ve PCA ile boyut-eşitlemeli ablasyon analizleri ile ikili özellik seti kombinasyonları karşılaştırılmış, farkların istatistiksel anlamlılığı eşlenmiş bootstrap testi ile sınanmıştır. Ayrıca n-gram morfolojik örtüşme analizi ve sağlamlık sınamaları (minimum sözcük sayısı filtreleme ve metin sınırlandırma) gerçekleştirilmiştir.

Bulgular—Karakter n-gramları en yüksek tekil set performansına ulaşmıştır (F1 = 0.889 ± 0.049, Lojistik Regresyon); BERTurk 0.783 ± 0.081 ile ikinci sıradadır. Morfolojik özellikler Stanza ile 0.577, Zeyrek ile 0.491 değerine ulaşmıştır. Standart ablasyon, çoğu özellik setinin karakter n-gramları ile birleştirildiğinde ek katkı sağlamadığını göstermiştir. Ancak PCA ile boyut-eşitlemeli ablasyon, tekrarlı çapraz doğrulama (10×5) ile değerlendirildiğinde, karakter n-gramlarının sağlam ve istatistiksel olarak anlamlı bir katkı sağladığını (ΔF1 ≈ 0.07, p < 0.001); boyut eşitlendiğinde düşük-boyutlu stilometrik ve işlev sözcüğü setlerinin de anlamlı katkı verdiğini göstermiştir. Buna karşılık morfolojik setler yüksek-güçlü testte korunan bir katkı göstermemiştir. Bu sonuçlar özellik setleri arasında belirgin boyut farkı olduğunda standart ablasyonun yanıltıcı sonuçlar üretebileceğini göstermektedir. N-gram morfolojik örtüşme analizi en ayırt edici n-gramların %16'sının Türkçe ek dizilerine karşılık geldiğini ortaya koymuştur.

Sonuç—Standart ablasyonda morfolojik ve stilometrik özelliklerin etkisiz görünmesi kısmen özellik setleri arasındaki boyut dengesizliğini yansıtır. PCA ile boyut-eşitlemeli ablasyon, tekrarlı çapraz doğrulama ile değerlendirildiğinde, karakter n-gramları ile düşük-boyutlu stilometrik ve işlev sözcüğü setleri anlamlı katkı sağlarken morfolojik setler yüksek-güçlü testte korunmamıştır; tek bir 5-katlı koşudaki görünür katkıları yeniden üretilememiştir. Bu nedenle bitişken dillerde boyut dengesizliğini göz ardı eden ablasyon tasarımları yanıltıcı olabilir ve küçük derlemlerde anlamlılık iddiaları temkinli kurulmalıdır. Gelecek çalışmalarda derlemin genişletilmesi, yazar-konu karışmasının ele alınması ve çerçevenin farklı bitişken dillerde sınanması planlanmaktadır.

Anahtar Kelimeler

• Yazar tanıma
• Türkçe doğal dil işleme
• karakter n-gramları
• morfolojik özellikler
• boyut-eşitlemeli ablasyon
• BERTurk

Authorship attribution in Turkish short fiction: A comparative analysis of linguistic and computational feature sets

Öz

Context—Authorship attribution is the process of identifying the author of a text based on stylistic features. While methods developed for non-agglutinative languages such as English have matured, agglutinative languages like Turkish remain underexplored. The agglutinative structure of Turkish creates a distinct feature space that is not addressed by English-centric methods.

Objective—This study systematically compares six feature sets across five classifiers for authorship attribution in Turkish short stories. The feature sets are stylometric, morphological (Stanza UD and Zeyrek), function words, character n-grams (TF-IDF), and BERTurk sentence embeddings. To prevent dimensionality imbalance across feature sets from distorting ablation results, a PCA-based dimension-equalized ablation framework is proposed.

Method—A balanced corpus of 110 short stories (135431 words) was compiled from 11 stories by each of 10 authors. Six feature sets were extracted and five classifiers (SVM-Linear, SVM-RBF, Logistic Regression, Random Forest, MLP) were evaluated using 5-fold stratified cross-validation. Standard and PCA-based dimension-equalized ablation analyses were conducted to compare pairwise feature set combinations, and the statistical significance of differences was assessed using paired bootstrap testing. Additionally, n-gram morphological overlap analysis and robustness tests (minimum word count filtering and text truncation) were performed.

Results—Character n-grams achieved the highest individual set performance (F1 = 0.889 ± 0.049, Logistic Regression); BERTurk ranked second with 0.783 ± 0.081. Morphological features reached 0.577 with Stanza and 0.491 with Zeyrek. Standard ablation showed that most feature sets did not provide additional contribution when combined with character n-grams. Under PCA-based dimension-equalized ablation evaluated with repeated cross-validation (10×5), character n-grams made a robust, statistically significant contribution (ΔF1 ≈ 0.07, p < 0.001), and low-dimensional stylometric and function-word feature sets also contributed significantly once dimensionality was equalized; in contrast, the morphological feature sets showed no contribution that survived the higher-powered test. N-gram morphological overlap analysis showed that 16% of the most discriminative n-grams correspond to Turkish suffix sequences.

Conclusion—The apparent ineffectiveness of morphological and stylometric features in standard ablation partly reflects dimensionality imbalance across feature sets. Under PCA-based dimension-equalized ablation with repeated cross-validation, character n-grams and the low-dimensional stylometric and function-word sets contribute significantly, whereas the morphological sets do not survive the higher-powered test; their apparent effect in a single 5-fold run did not reproduce. Ablation designs ignoring dimensionality imbalance in agglutinative languages may thus mislead, and significance claims on small corpora warrant caution. Future work includes expanding the corpus, addressing author-topic confounding, and testing other agglutinative languages.

Anahtar Kelimeler

• Authorship attribution
• Turkish NLP
• character n-grams
• morphological features
• dimension-equalized ablation
• BERTurk

Kaynakça

1. E. Stamatatos, “A survey of modern authorship attribution methods”, Journal of the American Society for Information Science and Technology, 60(3), 538–556, 2009. https://doi.org/10.1002/asi.21001.
2. M. Koppel, J. Schler, S. Argamon, “Computational methods in authorship attribution”, Journal of the American Society for Information Science and Technology, 60(1), 9–26, 2009. https://doi.org/10.1002/asi.20961.
3. X. He, A. H. Lashkari, N. Vombatkere, D. P. Sharma, “Authorship attribution methods, challenges, and future research directions: A comprehensive survey”, Information, 15(3), 131, 2024. https://doi.org/10.3390/info15030131.
4. B. Huang, C. Chen, K. Shu, “Authorship attribution in the era of LLMs: Problems, methodologies, and challenges”, arXiv, 2024, https://arxiv.org/abs/2408.08946.
5. J. Burrows, “‘Delta’: a measure of stylistic difference and a guide to likely authorship”, Literary and Linguistic Computing, 17(3), 267–287, 2002. https://doi.org/10.1093/llc/17.3.267.
6. M. Kestemont, “Function words in authorship attribution: From black magic to theory?”, Proceedings of the 3rd Workshop on Computational Linguistics for Literature (CLFL), Gothenburg, Sweden, April 2014, 59–66. https://doi.org/10.3115/v1/w14-0908.
7. U. Sapkota, S. Bethard, M. Montes, T. Solorio, “Not all character n-grams are created equal: A study in authorship attribution”, Proceedings of the 2015 conference of the North American chapter of the association for computational linguistics: Human language Technologies (NAACL), Denver, USA, May-June 2015, 93–102. https://doi.org/10.3115/v1/n15-1010.
8. E. Stamatatos, “On the robustness of authorship attribution based on character n-gram features”, Journal of Law and Policy, 21(2), 421–439, 2013.

9. M. Fabien, E. Villatoro-Tello, P. Motlicek, S. Parida, “BertAA: BERT fine-tuning for authorship attribution”, Proceedings of the 17th International Conference on Natural Language Processing (ICON), Patna, India, December 2020, 127–137. https://aclanthology.org/2020.icon-main.16/.
10. C. Schöch, J. Dudar, E. Fileva, A. Šeļa, “Multilingual Stylometry: The Influence of Language on the Performance of Authorship Attribution using Corpora from the European Literary Text Collection (ELTeC)”, Proceedings of the Computational Humanities Research Conference 2024 (CHR 2024), Aarhus, Denmark, December 2024, 386-408, https://ceur-ws.org/Vol-3834/paper9.pdf.
11. J. Rybicki, M. Eder, “Deeper Delta across genres and languages: Do we really need the most frequent words?”, Literary and Linguistic Computing, 26(3), 315–321, 2011. https://doi.org/10.1093/llc/fqr031.
12. S. Evert, T. Proisl, F. Jannidis, I. Reger, S. Pielström, C. Schöch, T. Vitt, "Understanding and explaining Delta measures for authorship attribution", Digital Scholarship in the Humanities, 32(suppl_2), ii4-ii16, 2017. https://doi.org/10.1093/llc/fqx023.
13. K. Oflazer, “Turkish and its challenges for language processing”, Language Resources and Evaluation, 48(4), 639–653, 2014. https://doi.org/10.1007/s10579-014-9267-2.
14. Ç. Çöltekin, "A freely available morphological analyzer for Turkish", Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta, May 2010, 820-827. https://doi.org/10.63317/3sqjjmc7u7w5.
15. K. Oflazer, M. Saraçlar, Turkish Natural Language Processing, Cham, Switzerland, Springer, 2018.
16. H. Sak, T. Güngör, M. Saraçlar, “Turkish language resources: Morphological parser, morphological disambiguator and web corpus”, Proceedings of the 6th International Conference on Natural Language Processing (GoTAL 2008), Gothenburg, Sweden, 25-27 August 2008, 417-427. https://doi.org/10.1007/978-3-540-85287-2_40.
17. L. Johanson, “Turkic indirectives”, Evidentials: Turkic, Iranian and Neighbouring Languages, L. Johanson, B. Utas, Eds., Berlin, Germany, Mouton de Gruyter, 2000, 61-88.
18. A. Göksel, C. Kerslake, Turkish: A Comprehensive Grammar, London, UK, Routledge, 2005.
19. M. Haspelmath, E. König, Converbs in Cross-Linguistic Perspective: Structure and Meaning of Adverbial Verb Forms, Berlin, Germany, Mouton de Gruyter, 1995.
20. R. Gorman, “Author identification of short texts using dependency treebanks without vocabulary”, Digital Scholarship in the Humanities, 35(4), 812–825, 2020. https://doi.org/10.1093/llc/fqz070.
21. M. F. Amasyalı, B. Diri, “Automatic Turkish text categorization in terms of author, genre and gender”, Proceedings of the 11th International Conference on Applications of Natural Language to Information Systems, Klagenfurt, Austria, 31 May -02 June 2006, 221-226. https://doi.org/10.1007/11765448_22.
22. F. Türkoğlu, B. Diri, M. F. Amasyalı, “Author attribution of Turkish texts by feature mining”, Proceedings of the 3rd International Conference on Intelligent Computing (ICIC 2007), Qingdao, China, 21-24 August 2007, 1086–1093. https://doi.org/10.1007/978-3-540-74171-8_110.
23. Y. Bay, E. Çelebi, “Feature selection for enhanced author identification of Turkish text”, Proceeding of the 30th International Symposium on Computer and Information Sciences (ISCIS), London, England, 21-24 September 2015, 371-379. https://doi.org/10.1007/978-3-319-22635-4_34.
24. M. Güllü, H. Polat, "Text authorship identification based on ensemble learning and genetic algorithm combination in Turkish text", Politeknik Dergisi, 25(3), 1287–1297, 2022. https://doi.org/10.2339/politeknik.992493.
25. A. E. Pirli, “POS-tags, lemmatization, and feature frequencies: Stylometric analyses in Turkish using stylo”, International Journal of Digital Humanities, 6(3), 313–334, 2025. https://doi.org/10.1007/s42803-024-00094-1.
26. N. Ş. Saygılı, T. Acarman, T. Amghar, B. Levrat, “Taking Advantage of Turkish Characteristic Features to Tackle with Authorship Attribution Problems for Turkish”, Proceedings of ICCGI 2016: The Eleventh International Multi-Conference on Computing in the Global Information Technology, Barcelona, Spain, November 2016, 25–29.
27. H. Gündüz, “Scalable gender profiling from Turkish texts using deep embeddings and meta-heuristic feature selection”, Journal of Theoretical and Applied Electronic Commerce Research, 20(4), 253, 2025. https://doi.org/10.3390/jtaer20040253.
28. S. Yıldırım, T. Yıldız, “Türkçe için karşılaştırmalı metin sınıflandırma analizi”, Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi, 24(5), 879-886, 2018. https://doi.org/10.5505/pajes.2018.15931.
29. F. Can, J. M. Patton, “Change of word characteristics in 20th-century Turkish literature: A statistical analysis”, Journal of Quantitative Linguistics, 17(3), 167–190, 2010. https://doi.org/10.1080/09296174.2010.485444.
30. K. Altıntaş, F. Can, J. M. Patton, “Language change quantification using time-separated parallel translations”, Literary and Linguistic Computing, 22(4), 375–393, 2007. https://doi.org/10.1093/llc/fqm026.
31. M. Eder, “Does size matter? Authorship attribution, small samples, big problem”, Digital Scholarship in the Humanities, 30(2), 167–182, 2015. https://doi.org/10.1093/llc/fqt066.
32. K. Kettunen, “Can type-token ratio be used to show morphological complexity of languages?”, Journal of Quantitative Linguistics, 21(3), 223–245, 2014. https://doi.org/10.1080/09296174.2014.911506.
33. G. U. Yule, The Statistical Study of Literary Vocabulary, Cambridge, UK, Cambridge University Press, 1944.
34. P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Manning, “Stanza: A Python natural language processing toolkit for many human languages”, Proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations, Online, July 2020, 101–108. https://doi.org/10.18653/v1/2020.acl-demos.14.
35. O. Bulat, “Zeyrek: Morphological analyzer and lemmatizer for Turkish (Python port of Zemberek-NLP)”, GitHub repository, https://github.com/obulat/zeyrek (26.04.2026).
36. A. A. Akın, M. D. Akın, "Zemberek, an open source NLP framework for Turkic languages", https://github.com/ahmetaa/zemberek-nlp.
37. S. Schweter, “BERTurk – BERT models for Turkish”, Zenodo, 2020, https://doi.org/10.5281/zenodo.3770924.
38. E. Çelik, "bert-base-turkish-cased-mean-nli-stsb-tr: Türkçe cümle-dönüştürücü (sentence-transformers) modeli", Hugging Face Model Hub, https://huggingface.co/emrecan/bert-base-turkish-cased-mean-nli-stsb-tr (23.06.2026).
39. C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”, Nature Machine Intelligence, 1(5), 206–215, 2019. https://doi.org/10.1038/s42256-019-0048-x.