İkinci dil (D2) İngilizce metinlerde yazma kalitesini etkileyen dilbilimsel faktörler nelerdir? Kapsamlı bir yaklaşım

Abstract

Öğrenci derlemi araştırmalarındaki güncel eğilimler, derlemlerin doğal dil işleme (DDİ) teknikleriyle bütünleştirilmesinin pedagojik açıdan daha elverişli sonuçlar verebileceğini göstermektedir. Bu araştırma alanındaki çalışmalar, bugüne kadar İkinci Dil (D2) İngilizce yazma kalitesini farklı açılardan incelemiş ve çeşitli dilbilimsel özellikleri titizlikle ele almıştır. Ancak, yalnızca sınırlı sayıda çalışma farklı dilbilimsel özellikleri çok boyutlu olarak analiz etmiş ve hiçbir çalışma çeşitli dilbilimsel özelliklerin Türkiye bağlamındaki D2 İngilizce yazma kalitesi üzerindeki etkisini araştırmamıştır. Önceki bulgulardan hareketle ve Türkiye bağlamındaki bu boşluğu göz önünde bulundurarak hazırlanan bu çalışma; dilbilimsel özellikleri belirlemek için özelleşmiş bir öğrenci derlemi ile DDİ araçlarını kullanmakta ve D2 yazma kalitesini etkileyen çok yönlü dilbilimsel faktörleri incelemeyi amaçlamaktadır. Bulgular; belirli dilbilimsel ve metinsel özelliklerin (örneğin; word2vec, akademik sözcük sıklığı, öbeksel sıklık ve dilbilgisi hataları) D2 İngilizce yazma kalitesinde önemli bir rol oynayabileceğini göstermektedir. Karma etkiler modeli sonuçları, önceki araştırmalar ve kullanım temelli dilbilimsel modeller ışığında tartışılmaktadır.

Keywords

• Öğrenci Derlemi
• Doğal Dil İşleme
• Dilbilimsel Özellikler
• Yazma Kalitesi
• D2'de Yazma

Supporting Institution

Türkiye Bilimsel ve Teknolojik Araştırma Kurumu

Project Number

220K289

Ethical Statement

Ankara Üniversitesi Etik Kurulunun 209/03/2021 tarihli toplantısı 3/9 sayılı kararı

What linguistic factors affect writing quality in L2 English writing? A multidimensional approach

Abstract

Current trends in learner corpus research indicate that integrating corpora with natural language processing (NLP) techniques can yield pedagogically more convenient results. Studies in this line of research to date have investigated writing quality in L2 English from different angles and scrutinised various linguistic features. However, only few studies have analysed different linguistic features multi- dimensionally and no study has explored the effect of various linguistic features on L2 English writing quality in the Turkish context. Building on earlier findings and considering the niche in the Turkish context, the current study uses a specialised learner corpus and NLP tools to extract linguistic features and attempts to investigate the multi-faceted linguistic factors influencing writing quality in L2. The findings suggest that certain linguistic and textual features (e.g., word2vec, academic word frequency, phrasal frequency, and grammar errors) can play an important role in writing quality in L2 English. The mixed effects model results are discussed considering previous research and usage-based linguistic models.

Keywords

• Learner Corpus
• Natural Language Processing
• Linguistic Features
• Writing Quality
• Writing in L2

Supporting Institution

The Scientific and Technological Research Council of Türkiye

Project Number

220K289

Ethical Statement

Ankara University Ethics Committee (29/03/2021) Decision number: 3/9

References

1. Abdi Tabari, M., & Johnson, M. D. (2023). Exploring new insights into the role of cohesive devices in written academic genres. Assessing Writing, 57, 100749. https://doi.org/https://doi.org/10.1016/j.asw.2023.100749
2. Altenberg, B. (1991). ‘A bibliography of publications relating to English computer corpora’. In S. Johansson & A. B. Stenstrom (eds.). English computer corpora: Selected papers and research guide. New York: Mouton, 355-396.
3. Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. Journal of Technology, Learning, and Assessment, 4(3). Available from http://www.jtla.org
4. Barker, F., Salamoura, A. & Saville, N. (2015). "Learner corpora and language testing". In S. Granger, G. Gilquin & F. Meunier, Eds., The Cambridge Handbook of Learner Corpus Research, Cambridge Handbooks in Language and Linguistics. Cambridge University Press, 511–534.
5. Berman, R. and Verhoeven, L. (2002). Cross-linguistic perspectives on the development of text-production abilities: Speech and writing. Written Language and Literacy, 5 (1), 1-43. https://doi.org/10.1075/wll.5.1
6. Bestgen, Y. (2017). Beyond single-word measures: L2 writing assessment, lexical richness and formulaic competence. System, 69, 65–78. https://doi.org/10.1016/j.system.2017.08.004
7. Clandfield, L., Goldstein, B., Jones, C., & Kerr, P. (2019). Evolve: Student's Book. Cambridge University Press. Council of Europe (2020). Common European Framework of Reference for Languages: Learning, Teaching, Assessment—Companion Volume. Council of Europe Publishing. https://www.coe.int/en/web/common-european-framework-reference-languages
8. Crossley, S. A. (2020). Linguistic features in writing quality and development: An overview. Journal of Writing Research, 11(3), 415–443. https://doi.org/10.17239/JOWR-2020.11.03.01

9. Crossley, S. A., Bradfield, F., & Bustamante, A. (2019a). Using human judgments to examine the validity of automated grammar, syntax, and mechanical errors in writing. Journal of Writing Research, 11(2), 251-270. https://doi.org/10.17239/jowr-2019.11.02.01
10. Crossley, S. A., Kyle, K., & Dascalu, M. (2019b). The tool for the automatic analysis of cohesion 2.0: Integrating semantic similarity and text overlap. Behavior Research Methods, 51(1), 14-27. doi: https://doi.org/10.3758/s13428-018-1142-4
11. Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 32, 1-16. https://doi.org/10.1016/j.jslw.2016.01.003
12. Crossley, S. A., Kyle, K., Varner, L., Gou, L., & McNamara, D. S. (2014). Linguistic microfeatures to predict L2 writing proficiency: A case study in automated writing evaluation. Journal of Writing Assessment, 7(1).
13. Crossley, S. A., & McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In S. Ohlsson & R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 984-989). Austin, TX: Cognitive Science Society.
14. Crossley, S. A., & McNamara, D. S. (2011). Text coherence and judgments of essay quality: Models of quality and coherence. In L. Carlson, C. Hoelscher, & T. F. Shipley (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society. (pp. 1236-1241). Austin, TX: Cognitive Science Society.
15. Crossley, S. A., & McNamara, D. S. (2012). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35 (2), 115-135. https://doi.org/10.1111/j.1467-9817.2010.01449.x
16. Crossley, S. A., & McNamara, D. (2013). Applications of text analysis tools for spoken response grading. Language Learning & Technology, 17(2), 171–192.
17. Crossley, S. A., Salsbury, T., & McNamara, D. S. (2012). Predicting the proficiency level of language learners using lexical indices. Language Testing, 29(2), 243–263. https://doi. org/10.1177/0265532211419331
18. Dalkey, N., & Helmer, O. (1963.) An Experimental Application of the Delphi Method to the Use of Experts. Management Science, 9, 458-467.
19. Du, X., Afzaal, M., & Al Fadda, H. (2022). Collocation use in EFL learners’ writing across multiple language proficiencies: A corpus-driven study. Front. Psychol. 13:752134. doi: 10.3389/fpsyg.2022.752134
20. Durrant, P., Moxley, J., & McCallum, L. (2019). Vocabulary sophistication in First-Year Composition assignments. International Journal of Corpus Linguistics, 24(1), 33–66. https://doi.org/10.1075/ijcl.17052.dur
21. Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185. https://doi.org/10.1177/0265532207086780
22. Ellis, N. C., & Wulff, S. (2020). Usage-based approaches to SLA. For: Bill VanPatten, Gregory Keating and Stefanie Wulff (eds.), Theories in second language acquisition. 3rd edition. New York: Routledge, 63-82.
23. Engber, C. A. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4(2), 139-155. https://doi.org/10.1016/1060-3743(95)90004-7
24. Field, A., (2016). An Adventure in Statistics: The Reality Enigma. Quantitative/Statistical Research. SAGE Publications, London ISBN 9781446210451.
25. Francis, W. N., & H. Kučera. (1964). A Standard Corpus of Present-Day Edited American English, for use with Digital Computers. Providence, Rhode Island: Department of Linguistics, Brown University. Revised 1971. Revised and amplified 1979.
26. Freedman, D. A. (1983). A note on screening regression equations. The American Statistician 37: 152–155.
27. Gallucci, M. (2019). GAMLj: General analyses for linear models. [jamovi module]. Retrieved from https://gamlj.github.io/.
28. Gallucci, M. (2020). Model goodness of fit in GAMLj. . link.
29. Garner, J. (2022). The cross-sectional development of verb–noun collocations as constructions in L2 writing. International Review of Applied Linguistics in Language Teaching, 60(3), 909-935. https://doi.org/10.1515/iral-2019-0169
30. Geertzen, J., Alexopoulou, T., & Korhonen, A. (2014). Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCAMDAT). In Proceedings of the 31st second language research forum (SLRF). Cascadilla Press
31. Gilquin, G. (2023). Written learner corpora to inform teaching. In The Routledge Handbook of Corpora and English Language Teaching and Learning (pp. 281-295). Routledge.
32. Goldberg, A. (2006). Constructions at Work: The Nature of Generalization in Language. New York: Oxford University Press. Cognitive Linguistics Bibliography (CogBib). Retrieved 2023-04-06, from https://www.degruyter.com/database/COGBIB/entry/cogbib.4775/html.
33. Granger, S. (1998). Prefabricated patterns in advanced EFL writing collocations and formulae. Phraseology: Theory, Analysis and Applications. Oxford: Clarendon Press.
34. Granger, S. (2009). The contribution of learner corpora to second language acquisition and foreign language teaching: A critical evaluation. In K. Aijmer (Ed.), Corpora and language teaching (pp. 13-32). John Benjamins. https://doi.org/10.1075/scl.33.04gra
35. Granger, S., & Bestgen, Y. (2014). The use of collocations by intermediate vs. advanced non-native writers: A bigram-based study. International Review of Applied Linguistics, 52, 229–252. https://doi.org/10.1515/iral-2014-0011
36. Grant, L., & Ginther, A. (2000). Using Computer-Tagged Linguistic Features to Describe L2 Writing Differences. Journal of Second Language Writing, 9(2), 123–145. https://doi.org/https://doi.org/10.1016/S1060-3743(00)00019-9
37. Gries, S., Hampe, B., & Schönefeld, D. (2005). Converging evidence: Bringing together experimental and corpus data on the association of verbs and constructions. Cognitive Linguistics, 16(4), 635-676. https://doi.org/10.1515/cogl.2005.16.4.635
38. Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218-238. https://doi.org/10.1016/j.asw.
39. Hwang, H., & Kim, H. (2023). Automatic analysis of constructional diversity as a predictor of EFL students’ writing proficiency. Applied Linguistics, 44(1), 127–147. https://doi.org/10.1093/applin/amac046
40. Jarvis, S. (2002). Short texts, best-fitting curves and new measures of lexical diversity. Language Testing, 19(1), 57-84. https://doi.org/10.1191/0265532202lt220oa
41. Jarvis, S., Grant, L., Bikowski, D., & Ferris, D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12(4), 377-403. https://doi.org/10.1016/j.jslw.2003.09.001
42. Kim, M., Crossley, S. A., & Kyle, K. (2018). Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, 102(1), 120-141. https://doi.org/10.1111/modl.12447
43. Koo T. K., & Li M. Y. (2016). A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J Chiropr Med, 15(2):155-63. doi: 10.1016/j.jcm.2016.02.012. Epub 2016 Mar 31. Erratum in: J Chiropr Med. 2017 Dec;16(4):346. PMID: 27330520; PMCID: PMC4913118.
44. Kuiken, F., & Vedder, I. (2019). Syntactic complexity across proficiency and languages: L2 and L1 writing in Dutch, Italian and Spanish. International Journal of Applied Linguistics, 29(2), 192–210. https://doi.org/https://doi.org/10.1111/ijal.12256
45. Kyle, K. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication [Doctoral dissertation, Georgia State University]. ScholarWorks @ Georgia State University. http://scholarworks.gsu.edu/alesl_diss/35
46. Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly 49(4), 757-786. https://doi.org/10.1002/tesq.194
47. Kyle, K., & Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34, 12-24. https://doi.org/10.1016/j.jslw.2016.10.003
48. Kyle, K., & Crossley, S. (2017). Assessing syntactic sophistication in L2 writing: A usage-based approach. Language Testing, 34(4), 513-535. https://doi.org/10.1177/0265532217712554
49. Kyle, K., & Crossley, S. A. (2018). Measuring Syntactic Complexity in L2 Writing Using Fine-Grained Clausal and Phrasal Indices. The Modern Language Journal, 102(2), 333–349. https://doi.org/https://doi.org/10.1111/modl.12468
50. Kyle, K., Crossley, S. A., & Berger, C. (2018). The tool for the analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods 50(3), pp. 1030-1046. https://doi.org/10.3758/s13428-017-0924-4
51. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240. https://doi.org/10.1037/0033-295X.104.2.211
52. Lemhöfer, K., & Broersma, M. (2012). Introducing LexTALE: A quick and valid Lexical Test for Advanced Learners of English. Behavior Research Methods, 44, 325-343.
53. Li, P., Zhang, F., Yu, A., & Zhao, X. (2020). Language History Questionnaire (LHQ3): An enhanced tool for assessing multilingual experience. Bilingualism: Language and Cognition, 23(5), 938-944. doi:10.1017/S1366728918001153
54. Lu, X. (2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 45 (1), 36-62. https://doi.org/ 10.5054/tq.2011.240859
55. Lu, X., & Ai, H. (2015). Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds. Journal of Second Language Writing, 29, 16–27. https://doi.org/https://doi.org/10.1016/j.jslw.2015.06.003
56. Lu, X., & Hu, R. (2022). Sense-aware lexical sophistication indices and their relationship to second language writing quality. Behavior Research Methods, 54(3), 1444-1460. https://doi.org/10.3758/s13428-021-01675-6
57. Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208. https://doi.org/10.3758/BF03204766
58. MacArthur, C. A., Jennings, A., & Philippakos, Z. A. (2019). Which linguistic features predict quality of argumentative writing for college basic writers, and how do those features change with instruction?. Reading and Writing, 32(6), 1553-1574. https://doi.org/10.1007/s11145-018-9853-6
59. McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural Language Processing in an Intelligent Writing Strategy Tutoring System. Behavior Research Methods, 45(2), 499-515. https://doi.org/10.3758/s13428-012-0258-1
60. Menard, S. (2002). Applied logistic regression analysis. SAGE Publications, Inc., https://doi.org/10.4135/9781412983433
61. Meunier, F. (2021). Introduction to Learner Corpus Research. In N. Tracy-Ventura & M. Paquot (Eds.), The Routledge Handbook of Second Language Acquisition and Corpora (pp. 23–36). Routledge.
62. Mikolov, T., Chen, K., Corrado, G. S., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. International Conference on Learning Representations.
63. Monteiro, K. R., Crossley, S. A., & Kyle, K. (2020). In search of new benchmarks: Using L2 lexical frequency and contextual diversity indices to assess second language writing. Applied Linguistics, 41(2), 280-300. https://doi.org/10.1093/applin/amy056
64. Monteiro, K., Crossley, S., Botarleanu, R.-M., & Dascălu, M. (2023). L2 and L1 semantic context indices as automated measures of lexical sophistication. Language Testing, 026553222211479. https://doi.org/10.1177/02655322221147924
65. Myers, R.H. (1990) Classical and modern regression with applications. PWS-Kent Publishing, Boston.
66. Myles, F. (2015). Second language acquisition theory and learner corpus research. In Granger, S., Gilquin, G., & Meunier, F. (eds.), Cambridge Handbook of Learner Corpus Research (pp. 309-331). Cambridge University Press. https://doi.org/10.1017/CBO9781139649414.014
67. Myles, F. (2021). Commentary: An SLA Perspective on Learner Corpus Research. In B. Le Bruyn & M. Paquot (eds.), Learner Corpus Research Meets Second Language Acquisition (pp. 258-273). Cambridge University Press. https://doi.org/10.1017/9781108674577.013
68. Norris, J. M., & Ortega, L. (2009). Towards an Organic Approach to Investigating CAF in Instructed SLA: The Case of Complexity. Applied Linguistics, 30(4), 555–578. https://doi.org/10.1093/applin/amp044
69. Ortega, L. (2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24, doi:10.1093/applin/24.4.492.
70. Öksüz, D., Brezina, V., & Rebuschat, P. (2021). Collocational processing in L1 and L2: The effects of word Frequency, collocational frequency, and association. Language Learning, 71(1), 55–98. https://doi.org/10.1111/lang.12427
71. Paquot, M., & Plonsky, L. (2017). Quantitative research methods and study quality in learner corpus research. International Journal of Learner Corpus Research, 3(1), 61–94. doi:10.1075/ijlcr.3.1.03paq
72. Park, K. (2014). Corpora and language assessment: The state of the art. Language Assessment Quarterly, 11(1), 27e44. http://dx.doi.org/10.1080/15434303. 2013.872647