An Evaluation of Syntactic Complexity, Lexical Diversity and Text Length as Predictors of Foreign Language Writing Scores

Year 2024, Volume: 26 Issue: 4, 542 - 553, 31.12.2024

Zafer Susoy , Gül Durmuşoğlu Köse

https://doi.org/10.17556/erziefd.1483282

Abstract

The main premise of this study is to investigate to what extent syntactic complexity (SC), lexical diversity (LD), and text length (TL) correlate to foreign language (FL) writing quality scores assigned by human judges for the English essays of 204 pre-service teachers of English of two different curricular levels (first and fourth-year students). The study adopts a sequential-explanatory mixed-method research design. To that end, eight instructors rating student papers for 16 years on average were interviewed. The statistical analyses reveal that the 4th-year students outperformed the 1st-year students in TL, writing scores, and five indices of SC and LD. Subsequent regression analyses explained the variance in overall writing scores. The qualitative results showed variability in the instructors' ability to detect and prioritize these linguistic features, showing that while some instructors had a nuanced understanding of SC and LD, others emphasized overall content and organization more than linguistic complexity. The role of syntactic complexity, linguistic diversity, and text length as predictors of foreign language writing quality revealed that while human raters recognize these linguistic features to varying extents, their evaluation can be enhanced through standardized assessment practices and the integration of automated tools.

Keywords

Composition, proficiency, writing assessment, English as a second foreign language

Sözdizimsel Karmaşıklık, Sözcük Çeşitliliği ve Metin Uzunluğunun Yabancı Dil Yazma Puanlarının Belirleyicisi Olarak Değerlendirilmesi

Abstract

Bu çalışmanın ana amacı, İngilizce deneme yazılarının değerlendirilmesi için insan hakemler tarafından verilen yabancı dil yazma kalite puanlarını ne ölçüde cümle yapısı karmaşıklığı (CYK), kelime çeşitliliği (KC) ve metin uzunluğu (MU) öngörebileceğini incelemektir. Çalışma, ardışık-açıklayıcı karma yöntem araştırma tasarımını benimser. Bu amaçla, ortalama 16 yıl öğrenci yazılarını değerlendiren 8 eğitmenle görüşmeler yapılmıştır. İstatistiksel analizler, 4. sınıf öğrencilerinin MU, yazma puanları ve CYK ve KC'nin toplam 5 göstergesinde 1. sınıf öğrencilerinden daha başarılı olduğunu ortaya koymuştur. Ardından yapılan regresyon analizleri, genel yazma puanlarının varyansını açıklamıştır. Nitel sonuçlar, eğitmenlerin genel puanlama prosedürüne ve CYK ve KC'yi puanlama sürecinde ne kadar iyi kavrayıp dikkate aldıklarına ilişkin içgörüler sağlamıştır.

Keywords

Kompozisyon, yeterlilik, yazma değerlendirmesi, ikinci yabancı dil olarak İngilizce

Ethical Statement

Dosyalara yüklenmiştir

Supporting Institution

yok

Thanks

yok

References

• Becker, A. (2010). Distinguishing linguistic and discourse features in ESL students’ written performance. Modern Journal of Applied Linguistics, 2, 406-424.
• Beers, S. F., and Nagy, W. E. (2009). Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre? Reading and Writing, 22(2), 185-200.
• Bi, P., and Jiang, J. (2020). Syntactic complexity in assessing young adolescent EFL learners’ writings: Syntactic elaboration and diversity. System, 91, 102248. doi.org/10.1016/j.system.2020.102248
• Bulté, B., and Housen, A. (2012). Defining and operationalizing L2 complexity. Dimensions of L2 performance and proficiency: Complexity, accuracy, and fluency in SLA, 32, 21.
• Bulté, B., and Housen, A. (2014). Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 26, 42-65. doi.org/10.1016/j.jslw.2014.09.005
• Casal, J. E., and Lu, X. (2021). ‘Maybe complicated is a better word’: Second-language English graduate student responses to syntactic complexity in a genre-based academic writing course. International Journal of English for Academic Purposes: Research and Practice, 2021(Spring), 95-115.
• Creswell, J. W. (2002). Educational research: Planning, conducting, and evaluating quantitative. Prentice Hall
• Crossley, S. A., and McNamara, D. S. (2010). Cohesion, coherence, and expert evaluations of writing proficiency. In S. Ohlsson and R. Catrambone (Eds.), Proceedings of the 32nd Annual Conference of the Cognitive Science Society (pp. 984-989). Austin, TX: Cognitive Science Society.
• Crossley, S. A., and McNamara, D. S. (2011). Understanding expert ratings of essay quality: Coh-Metrix analyzes first and second-language writing. International Journal of Continuing Engineering Education and Life Long Learning, 21(2-3), 170-191. doi.org/10.1504/IJCEELL.2011.040197
• Daller, H., R. Van Hout, and J. Treffers-Daller. (2003). ‘Lexical richness in the spontaneous speech of bilinguals,’ Applied Linguistics 24: 197–222. doi.org/10.1093/applin/24.2.197
• Evans, J.D. (1996) Straightforward statistics for the behavioral sciences. Pacific Grove, CA: Brooks/Cole Publishing
• Foster, P., and Skehan, P. (1996). The influence of planning and task type on second language performance. Studies in Second Language Acquisition, 18(3), 299-323. doi.org/10.1017/S0272263100015047
• Geiser, S., and Studley, R. (2001, October). UC and the SAT: Predictive validity and differential impact of the SAT I and SAT II at the University of California. Paper presented at the Meeting of the Board of Admissions and Relations with Schools of the University of California.
• Graesser, A. C., McNamara, D. S., and Kulikowich, J. M. (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40(5), 223–234 doi.org/10.3102/0013189X11413
• Guo, L., Crossley, S. A., and McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218-238. doi.org/10.1016/j.asw.2013.05.002
• Halliday, M. A. K., and Hassan, R. (1985). Language, context, and text: Aspect of language in a social semiotic perspective. Geelong, Australia: Deakin University Press.
• Harley, B., and King, M. L. (1989). Verb lexis in the written compositions of young L2 learners. Studies in Second Language Acquisition, 11, 415-439. https://doi.org/10.1017/S0272263100008421
• Hmelo, C., Holton, D., and Kolodner, J. (2000). Designing to Learn About Complex Systems. Journal of the Learning Sciences, 9, 247 - 298. https://doi.org/10.1207/S15327809JLS0903_2.
• Jarvis, S. (2002). Short texts, best-fitting curves, and new measures of lexical diversity. Language Testing, 19(1), 57-84. doi.org/10.1191/0265532202lt220oa
• Jarvis, S., Grant, L., Bikowski, D., and Ferris, D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12(4), 377-403. doi.org/10.1016/j.jslw.2003.09.001
• Jenkins, J. R., Johnson, E., and Hileman, J. (2004). When is reading also writing: Sources of individual differences on the new reading performance assessments. Scientific Studies of Reading, 8(2), 125-151. doi.org/10.1207/s1532799xssr0802_2
• Johansson, C., and Geisler, C. (2011). Syntactic aspects of the writing of Swedish L2 learners of English. Language and Computers-Studies in Practical Linguistics, 73(1), 139. doi.org/10.1163/9789401206884_009
• Laufer, B. and P. Nation. (1995). ‘Vocabulary size and use – lexical richness in L2 written production,’ Applied Linguistics 16: 307–22. doi.org/10.1093/applin/16.3.307
• Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language learning, 40(3), 387-417. doi.org/10.1111/j.1467-1770.1990.tb00669.x
• Linnarud, M. (1986). Lexis in composition: A performance analysis of Swedish learners' written English. Malmo, Sweden: Liber Forlag Malmo.
• Lu, X. (2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers' language development. Tesol Quarterly, 36-62. doi.org/10.5054/tq.2011.240859
• Lu, X. 2010. Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics 15(4): 474–496. doi.org/10.1075/ijcl.15.4.02lu
• Malvern, D. D., Richards, B. J., Chipere, N., and Durán, P. (2004). Lexical diversity and language development. Houndmills, Hampshire, UK: Palgrave Macmillan.
• Mazgutova, D., and Kormos, J. (2015). Syntactic and lexical development in an intensive English for Academic Purposes program. Journal of Second Language Writing, 29, 3-15. doi.org/10.1016/j.jslw.2015.06.004
• McNamara, D. S., Crossley, S. A., and McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27, 57-86.
• McNamara, D. S., Graesser, A. C., McCarthy, P. M., and Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge University Press.
• Muter, V., Hulme, C., Snowling, M., and Stevenson, J. (2004). Phonemes, rimes, vocabulary, and grammatical skills as foundations of early reading development: evidence from a longitudinal study. Developmental psychology, 40 5, 665-81. https://doi.org/10.1037/0012-1649.40.5.665.
• Norris, J. M., and Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: the case of complexity. Applied Linguistics, 30, 555–578. doi.org/10.1093/applin/amp044
• Olinghouse, N. G. and J. Wilson. (2013). ‘The relationship between vocabulary and writing quality in three genres,’ Reading and Writing 26: 45–65
• Ong, J., and Zhang, L. J. (2010). Effects of task complexity on fluency and lexical complexity in EFL students’ argumentative writing. Journal of Second Language Writing, 19, 218–233. doi.org/10.1016/j.jslw.2010.10.003
• Ong, J., and Zhang, L. J. (2013). Effects of manipulation of cognitive processes on English-as-a-foreign-language (EFL) writers’ text quality. TESOL Quarterly, 47, 375–398. doi.org/10.1002/tesq.55
• Ortega, L. (2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24, 492–518 doi.org/10.1093/applin/24.4.492
• Ortega, L. (2012). Interlanguage complexity: A construct in search of theoretical renewal. In B. Kortmann and B. Szmrecsanyi (Eds.), Linguistic complexity: Second language acquisition, indigenization, contact (pp. 127–155). Berlin: De Gruyter.
• Ortega, L. (2015). Syntactic complexity in L2 writing: Progress and expansion. Journal of Second Language Writing, 29, 82-94. doi.org/10.1016/j.jslw.2015.06.008
• Pallotti, G. (2015). A simple view of linguistic complexity. Second Language Research, 31(1), 117-134. doi.org/10.1177/0267658314536435
• Ravid, D. (2005). Emergence of linguistic complexity in later language development: Evidence from expository text construction. In Perspectives on language and language development (pp. 337-355). Springer, Boston, MA.
• Stockwell, G. and Harrington, M. (2003). The incidental development of L2 proficiency in NS-NNS email interactions. CALICO Journal 20(2): 337–359.
• Stockwell, G. (2005). Syntactical and lexical development in NNS-NNS asynchronous CMC. The JALT CALL Journal, 1(3), 33-49.
• Treffers-Daller, J., Parslow, P., and Williams, S. (2016). Back to basics: how measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics, 39(3), 302-327. doi.org/10.1093/applin/amw009
• Vyatkina, N., Hirschmann, H., and Golcher, F. (2015). Syntactic modification at early stages of L2 German writing development: A longitudinal learner corpus study. Journal of Second Language Writing, 29, 28-50. doi.org/10.1016/j.jslw.2015.06.006
• Weber, R. P. (1990). Basic content analysis (2nd ed.). Thousand Oaks, CA: Sage Publications
• Wolfe-Quintero, K., Inagaki, S., and Kim, H.-Y. (1998). Second language development in writing: Measures of fluency, accuracy, and complexity. Honolulu, HI: University of Hawaii Press.
• Yu, G. (2009). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31(2), 236-259. doi.org/10.1093/applin/amp024