Assessing Reading Proficiency in Turkish as a Foreign Language: A Validation, Skill and Standardisation Study

Abstract

Language assessment has long become an independent academic discipline, producing and contributing to a large body of literature. However, key considerations in testing and assessment in the context of Teaching Turkish as a Foreign Language (TTFL) has been often neglected so far, leading to exams that lack evidence for validation. The present study aims to address this shortcoming by offering a standardized validation approach for assessing reading skills in Turkish proficiency exams, using several theoretically robust validity frameworks and cognitive models. The study particularly focussed on the development and validation of B2-C1 reading proficiency descriptors using a mixed-methods approach with the participation of 8 field experts. The findings from the study can provide guidance to item writers in the design and development of standardized test specifications and test items compatible with the levels intended to be assessed.

Keywords

• teaching Turkish
• language assessment
• reading skills
• a priori validity

References

1. Alderson, J. Charles & et al. (1995) Language test construction and evaluation. Cambridge University Press,
2. Alderson, J. Charles & et al. (2006). Analysing tests of reading and listening in relation to the common European framework of reference: The experience of the Dutch CEFR construct project. Language Assesment Quarterly, 3(1), 3-30.
3. ALTE. The ALTE can do project: Articles and can do statements produced by the members of ALTE (1992-2002) (ALTECDP). (https://www.cambridgeenglish.org/images/28906-alte-can-do-document.pdf)
4. ALTE. (2011). Manual for language test development and examing: For use with the CEFR.
5. Altunkaya, Hatice (2016) Yabancı dil olarak Türkçe öğrenenlere yönelik okuduğunu anlama başarı testinin geliştirilmesi. [Development of a reading comprehension achievement test for learners of Turkish as a foreign language] Turkish Studies International Periodical for the Languages, Literature and History of Turkish or Turkic Volume 11/3 Winter 2016, 113-138.
6. Boylu, Emrah. (2019). Yabancılara Türkçe öğretiminde ölçme değerlendirme uygulamaları ve standart oluşturma. [Measurement assesment practise and standardization in teaching Turkish to foreigners.] Doktora Tezi. Çanakkale Onsekiz Mart Üniversitesi.
7. Boylu, Emrah. (2021). Kuramdan uygulamaya yabancı dil olarak Türkçe öğretiminde ölçme ve değerlendirme. [Assessment and evaluation in teaching Turkish as a foreign language from theory to practice.] Pegem.
8. Bachman, L. F., & Palmer, A. S. (1996). Language Testing in Practice. Oxford University Press.

9. Council of Europe The Language Policy Division. (2009). Relating language examinations to the common European framework of reference for languages: Learning, teaching, assessment (CEFR) a manual.
10. Council of Europe The Language Policy Division. (2018). Common European framework of reference for languages: Learning, teaching, assessment (CEFR) companion volume.
11. Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. Sage publications.
12. Davidson, Fred ve Lynch, Brian K. (2002). Testcraft: A teacher’s guide to writing and using language test specifications. Yale University Press.
13. Day, Richard. R. ve Park, Jeong-sug. (2005). Devoloping reading comprehension questions. Reading in a Foreign Language, vol.17, no. 1, ss. 60-73.
14. Durmuş, Mustafa. (2013). Türkçenin yabancılara öğretimi: Sorunlar, çözüm önerileri ve yabancılara Türkçe öğretiminin geleceğiyle ilgili görüşler. [Teaching Turkish to foreigners: Problems, solution suggestions and views on the future of teaching Turkish to foreigners.] Adıyaman Üniversitesi Sosyal Bilimler Enstitüsü Dergisi Türkçenin Eğitimi Öğretimi Özel Sayısı, Yıl 6, Sayı 11, Ocak, 207-228.
15. Duru, Hüseyin. (2014) Yabancı dil olarak Türkçe öğretiminde kullanılan soruların nitelik araştırması. [Qualitative research of questions used in teaching Turkish as a foreign language.] Doktora Tezi. Çanakkale Onsekiz Mart Üniversitesi.
16. Field, John. (2004). Psycholinguistics: The key concepts. Routledge.
17. Field, John. (2011). Cognitive validity. Examing speaking studies in language testing 30, (ed. Linda Taylor). Cambridge University Press, 65-111.
18. Fulcher, G. & Davidson, F. (2007). Language testing and assessment: An advanced resource book. Routledge.
19. Fulcher, G. (2010). Practical Language Testing. Routledge.
20. Frey, Bruce B. & et al. (2005). Item-writing rules: Collective wisdom. Teaching and teacher education, 21, 357-364.
21. Gedik, Enver. (2017). Yabancı dil olarak Türkçe öğretiminde ölçme ve değerlendirme. [Assesment and evlatuion in Teaching Turkish as a foreign language] Yüksek Lisans Tezi. İstanbul Arel Üniversitesi.
22. Green, Anthony. (2014). Exploring Language Assessment and Testing: Language in Action. Routledge.
23. Hughes, Arthur & Hughes, Jake. (2020) Testing for Language Teachers. Cambridge University Press.
24. Işıkoğlu, Metin. (2015) Yabancı Dil Olarak Türkçe Öğretiminde Kullanılan Yeterlik Sınavlarının Madde Yazımı Bakımından İncelenmesi (Mersin ve Sakarya Üniversiteleri Örneği). [Analysis of Proficiency Exams Developed for Teaching Turkish as A Foreign Language in Terms of Item Writing (Samples of Mersin and Sakarya Universities)] Yüksek Lisans Tezi. Atatürk Üniversitesi.
25. Jones, Neil. (2002) “Relating the ALTE Framework to the Common European Framework of Reference.” Case Studies on the use of the Common European Framework of Reference. ed. J. C. Alderson, Cambridge University Press,167-183.
26. Kane, M. (2006) “Validation”. Educational Measurement. ed. R. J. Brennan, 4, 18–64.
27. Keenan, Janice M. & et all. (2008) “Reading comprehension tests vary in the skills they assess: differential dependence on decoding and oral comprehension.” Scientific Studies of Reading, 12, 281– 300.
28. Khalifa, Hanan & Weir, Cyril. (2009) Examing Reading, research and practise in assessing second language reading. Cambridge University Press.
29. Kılıç, Selim. (2015) Kappa Testi. [Kappa Test] Journal of mood disorders. 5(3), 142-144.
30. Kırkgöz, Y. (2008). Globalization and English language policy in Turkey. Educational Policy, 22(5), 663-684.
31. Kintsch, Walter & van Dijk, Teun. (1978) “Toward a Model of Text Comprehension and Production.” Psychological Review, 85, 363-394.
32. Kintsch, Walter. (2005) “An Overview of Top-Down and Bottom-Up Effects in Comprehension: The CI Perspective.” Discourse Processes, 39(2&3),125–128.
33. Kintsch, Walter & Welsch. (2013) David M. “The construction-integration model: A framework for studying memory for text.” Relating Theory and Data, Psychology Press, 381-400.
34. Kolcuoğlu, Banu. (2022) Yabancı Dil Olarak Türkçe Öğretenlerin Ölçme ve Değerlendirme Okuryazarlıkları Üzerine Bir İnceleme. [An analysis on the assessment and evaluation literacy of teachers who teach Turkish as a foreign language] Yüksek Lisans Tezi. İstanbul Aydın Üniversitesi.
35. Mislevy, Robert J. & Haertel, Geneva, D. (2006) “Implications for evidence-centered design for educational assessment.” Educational Measurement: Issues and Practice, 25, 6–20.
36. Mislevy, Robert J. ve Sabatini, John P. (2012) “How research on reading and research on assessment are transforming reading assessment (or if they aren’t, how they ought to).” Measuring up: advances in how we assess reading ability. ed. J. Sabatini, E. Albro, & T. O'Reilly, Rowman & Littlefield Education,119–134.
37. North, B. (2007). The CEFR illustrative descriptor scales. Modern Language Journal, 91, 656-659.
38. Özyalçın, Kevser Esma. (2019) Yabancı Dil Olarak Türkçe Öğretenlerin Yenilenmiş Bloom Taksonomisine Göre Metin Altı Soru Yazma Becerileri Üzerine Bir Değerlendirme. [An evaluation of under the text question writing skills according to Bloom’s revised taxonomy of teachers of Turkish as a foreign language] Yüksek Lisans Tezi. Çanakkale Onsekiz Mart Üniversitesi.
39. Perfetti, Charles ve Adlof, Suzanne M. (2012) “Reading Comprehension: A Conceptual Framework from Word Meaning to Text Meaning”. Measuring up: advances in how we assess reading ability. ed. J. Sabatini, E. Albro, & T. O'Reilly, Rowman & Littlefield Education, 3-20.
40. Shin, Dong-il. (2012) “Item writing and writers.” The Routledge Handbook of Language Testing. ed. Glenn Fulcher, Fred Davidson, Routledge.
41. Spaan, Mary. (2006) “Test and Item Specifications Development.” Language Assesment Quarterly, 3(1), ss. 71-79.
42. TELC. (2013) Diller İçin Avrupa Ortak Öneriler Çerçevesi: Öğrenim, Öğretim ve Değerlendirme. [Common European framework of reference for languages: Learning, teaching, assessment].
43. Stromso, Helge I & Braten, Ivar. (2002) “Norwageian Law Students’ Use of Multiple Sources While Reading Expository Texts”. Reading Research Quarterly, vol. 37, no. 3; 208-227.
44. Ülper, Hakan. (2021) Okuma ve Anlamlandırma Becerilerinin Kazandırılması. [Acquiring Reading and Comprehension Skills] Nobel.
45. Ülper, Hakan ve diğerleri. (2017) Okuduğunu anlama Testinin Geliştirilmesi. [Developing reading Comprehension Test] Ahi Evran Üniversitesi Kırşehir Eğitim Fakültesi Dergisi (KEFAD), Cilt 18, Sayı 1, Nisan, 175-187.
46. Weir, Cyril. Language Testing and Validation: An Evidence-based Approach. (2005) Palgrave Macmillan.
47. Yüce, Sema Nur. (2022) Yabancı Dil Olarak Türkçe Öğretiminin Değerlendirilmesi: Öğretici ve Akademisyen Görüşleri. [Evaluation of teachıng turiısh as a foreign language: teachers' and academics' views] Yüksek Lisans Tezi, Ordu Üniversitesi.