Sözdizimsel Karmaşıklık, Sözcük Çeşitliliği ve Metin Uzunluğunun Yabancı Dil Yazma Puanlarının Belirleyicisi Olarak Değerlendirilmesi

Öz

Bu çalışmanın ana amacı, İngilizce deneme yazılarının değerlendirilmesi için insan hakemler tarafından verilen yabancı dil yazma kalite puanlarını ne ölçüde cümle yapısı karmaşıklığı (CYK), kelime çeşitliliği (KC) ve metin uzunluğu (MU) öngörebileceğini incelemektir. Çalışma, ardışık-açıklayıcı karma yöntem araştırma tasarımını benimser. Bu amaçla, ortalama 16 yıl öğrenci yazılarını değerlendiren 8 eğitmenle görüşmeler yapılmıştır. İstatistiksel analizler, 4. sınıf öğrencilerinin MU, yazma puanları ve CYK ve KC'nin toplam 5 göstergesinde 1. sınıf öğrencilerinden daha başarılı olduğunu ortaya koymuştur. Ardından yapılan regresyon analizleri, genel yazma puanlarının varyansını açıklamıştır. Nitel sonuçlar, eğitmenlerin genel puanlama prosedürüne ve CYK ve KC'yi puanlama sürecinde ne kadar iyi kavrayıp dikkate aldıklarına ilişkin içgörüler sağlamıştır.

Anahtar Kelimeler

• Kompozisyon
• yeterlilik
• yazma değerlendirmesi
• ikinci yabancı dil olarak İngilizce

An Evaluation of Syntactic Complexity, Lexical Diversity and Text Length as Predictors of Foreign Language Writing Scores

Abstract

The main premise of this study is to investigate to what extent syntactic complexity (SC), lexical diversity (LD), and text length (TL) correlate to foreign language (FL) writing quality scores assigned by human judges for the English essays of 204 pre-service teachers of English of two different curricular levels (first and fourth-year students). The study adopts a sequential-explanatory mixed-method research design. To that end, eight instructors rating student papers for 16 years on average were interviewed. The statistical analyses reveal that the 4th-year students outperformed the 1st-year students in TL, writing scores, and five indices of SC and LD. Subsequent regression analyses explained the variance in overall writing scores. The qualitative results showed variability in the instructors' ability to detect and prioritize these linguistic features, showing that while some instructors had a nuanced understanding of SC and LD, others emphasized overall content and organization more than linguistic complexity. The role of syntactic complexity, linguistic diversity, and text length as predictors of foreign language writing quality revealed that while human raters recognize these linguistic features to varying extents, their evaluation can be enhanced through standardized assessment practices and the integration of automated tools.

Keywords

• Composition
• proficiency
• writing assessment
• English as a second foreign language

Kaynakça

1. Becker, A. (2010). Distinguishing linguistic and discourse features in ESL students’ written performance. Modern Journal of Applied Linguistics, 2, 406-424.
2. Beers, S. F., and Nagy, W. E. (2009). Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre? Reading and Writing, 22(2), 185-200.
3. Bi, P., and Jiang, J. (2020). Syntactic complexity in assessing young adolescent EFL learners’ writings: Syntactic elaboration and diversity. System, 91, 102248. doi.org/10.1016/j.system.2020.102248
4. Bulté, B., and Housen, A. (2012). Defining and operationalizing L2 complexity. Dimensions of L2 performance and proficiency: Complexity, accuracy, and fluency in SLA, 32, 21.
5. Bulté, B., and Housen, A. (2014). Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 26, 42-65. doi.org/10.1016/j.jslw.2014.09.005
6. Casal, J. E., and Lu, X. (2021). ‘Maybe complicated is a better word’: Second-language English graduate student responses to syntactic complexity in a genre-based academic writing course. International Journal of English for Academic Purposes: Research and Practice, 2021(Spring), 95-115.
7. Creswell, J. W. (2002). Educational research: Planning, conducting, and evaluating quantitative. Prentice Hall
8. Crossley, S. A., and McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In S. Ohlsson and R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 984-989). Austin, TX: Cognitive Science Society.

9. Crossley, S. A., and McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh-Metrix analyzes first and second-language writing. International Journal of Continuing Engineering Education and Life Long Learning, 21(2-3), 170-191. doi.org/10.1504/IJCEELL.2011.040197
10. Daller, H., R. Van Hout, and J. Treffers-Daller. (2003). ‘Lexical richness in the spontaneous speech of bilinguals,’ Applied Linguistics 24: 197–222. doi.org/10.1093/applin/24.2.197
11. Evans, J.D. (1996) Straightforward statistics for the behavioral sciences. Pacific Grove, CA: Brooks/Cole Publishing
12. Foster, P., and Skehan, P. (1996). The influence of planning and task type on second language performance. Studies in Second Language Acquisition, 18(3), 299-323. doi.org/10.1017/S0272263100015047
13. Geiser, S., and Studley, R. (2001, October). UC and the SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California. Paper presented at the Meeting of the Board of Admissions and Relations with Schools of the University of California.
14. Graesser, A. C., McNamara, D. S., and Kulikowich, J. M. (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40(5), 223–234 doi.org/10.3102/0013189X11413
15. Guo, L., Crossley, S. A., and McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218-238. doi.org/10.1016/j.asw.2013.05.002
16. Halliday, M. A. K., and Hassan, R. (1985). Language, context, and text: Aspect of language in a social semiotic perspective. Geelong, Australia: Deakin University Press.
17. Harley, B., and King, M. L. (1989). Verb lexis in the written compositions of young L2 learners. Studies in Second Language Acquisition, 11, 415-439. https://doi.org/10.1017/S0272263100008421
18. Hmelo, C., Holton, D., and Kolodner, J. (2000). Designing to Learn About Complex Systems. Journal of the Learning Sciences, 9, 247 - 298. https://doi.org/10.1207/S15327809JLS0903_2.
19. Jarvis, S. (2002). Short texts, best-fitting curves, and new measures of lexical diversity. Language Testing, 19(1), 57-84. doi.org/10.1191/0265532202lt220oa
20. Jarvis, S., Grant, L., Bikowski, D., and Ferris, D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12(4), 377-403. doi.org/10.1016/j.jslw.2003.09.001
21. Jenkins, J. R., Johnson, E., and Hileman, J. (2004). When is reading also writing: Sources of individual differences on the new reading performance assessments. Scientific Studies of Reading, 8(2), 125-151. doi.org/10.1207/s1532799xssr0802_2
22. Johansson, C., and Geisler, C. (2011). Syntactic aspects of the writing of Swedish L2 learners of English. Language and Computers-Studies in Practical Linguistics, 73(1), 139. doi.org/10.1163/9789401206884_009
23. Laufer, B. and P. Nation. (1995). ‘Vocabulary size and use – lexical richness in L2 written production,’ Applied Linguistics 16: 307–22. doi.org/10.1093/applin/16.3.307
24. Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language learning, 40(3), 387-417. doi.org/10.1111/j.1467-1770.1990.tb00669.x
25. Linnarud, M. (1986). Lexis in composition: A performance analysis of Swedish learners' written English. Malmo, Sweden: Liber Forlag Malmo.
26. Lu, X. (2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers' language development. Tesol Quarterly, 36-62. doi.org/10.5054/tq.2011.240859
27. Lu, X. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics 15(4): 474–496. doi.org/10.1075/ijcl.15.4.02lu
28. Malvern, D. D., Richards, B. J., Chipere, N., and Durán, P. (2004). Lexical diversity and language development. Houndmills, Hampshire, UK: Palgrave Macmillan.
29. Mazgutova, D., and Kormos, J. (2015). Syntactic and lexical development in an intensive English for Academic Purposes program. Journal of Second Language Writing, 29, 3-15. doi.org/10.1016/j.jslw.2015.06.004
30. McNamara, D. S., Crossley, S. A., and McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86.
31. McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
32. Muter, V., Hulme, C., Snowling, M., and Stevenson, J. (2004). Phonemes, rimes, vocabulary, and grammatical skills as foundations of early reading development: evidence from a longitudinal study. Developmental psychology, 40 5, 665-81. https://doi.org/10.1037/0012-1649.40.5.665.
33. Norris, J. M., and Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: the case of complexity. Applied Linguistics, 30, 555–578. doi.org/10.1093/applin/amp044
34. Olinghouse, N. G. and J. Wilson. (2013). ‘The relationship between vocabulary and writing quality in three genres,’ Reading and Writing 26: 45–65
35. Ong, J., and Zhang, L. J. (2010). Effects of task complexity on fluency and lexical complexity in EFL students’ argumentative writing. Journal of Second Language Writing, 19, 218–233. doi.org/10.1016/j.jslw.2010.10.003
36. Ong, J., and Zhang, L. J. (2013). Effects of manipulation of cognitive processes on English-as-a-foreign-language (EFL) writers’ text quality. TESOL Quarterly, 47, 375–398. doi.org/10.1002/tesq.55
37. Ortega, L. (2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24, 492–518 doi.org/10.1093/applin/24.4.492
38. Ortega, L. (2012). Interlanguage complexity: A construct in search of theoretical renewal. In B. Kortmann and B. Szmrecsanyi (Eds.), Linguistic complexity: Second language acquisition, indigenization, contact (pp. 127–155). Berlin: De Gruyter.
39. Ortega, L. (2015). Syntactic complexity in L2 writing: Progress and expansion. Journal of Second Language Writing, 29, 82-94. doi.org/10.1016/j.jslw.2015.06.008
40. Pallotti, G. (2015). A simple view of linguistic complexity. Second Language Research, 31(1), 117-134. doi.org/10.1177/0267658314536435
41. Ravid, D. (2005). Emergence of linguistic complexity in later language development: Evidence from expository text construction. In Perspectives on language and language development (pp. 337-355). Springer, Boston, MA.
42. Stockwell, G. and Harrington, M. (2003). The incidental development of L2 proficiency in NS-NNS email interactions. CALICO Journal 20(2): 337–359.
43. Stockwell, G. (2005). Syntactical and lexical development in NNS-NNS asynchronous CMC. The JALT CALL Journal, 1(3), 33-49.
44. Treffers-Daller, J., Parslow, P., and Williams, S. (2016). Back to basics: how measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics, 39(3), 302-327. doi.org/10.1093/applin/amw009
45. Vyatkina, N., Hirschmann, H., and Golcher, F. (2015). Syntactic modification at early stages of L2 German writing development: A longitudinal learner corpus study. Journal of Second Language Writing, 29, 28-50. doi.org/10.1016/j.jslw.2015.06.006
46. Weber, R. P. (1990). Basic content analysis (2nd ed.). Thousand Oaks, CA: Sage Publications
47. Wolfe-Quintero, K., Inagaki, S., and Kim, H.-Y. (1998). Second language development in writing: Measures of fluency, accuracy, and complexity. Honolulu, HI: University of Hawaii Press.
48. Yu, G. (2009). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31(2), 236-259. doi.org/10.1093/applin/amp024