Striking the Balance between Validity and Reliability of a Listening Test in Turkish as a Second Language

Year 2017, Volume: 34 Issue: 1, 49 - 73, 17.12.2018

Emel Tozlu Aylin Ünaldı

Abstract

Evidence on the efficacy of an assessment tool is necessary in order to justify the decisions we make based on the scores from it. Validity evidence can be collected from several sources such as the stages before and after test administration. In the present research study, validity evidence of several types on a Turkish as a Second Language (TSL) Academic Listening Test is presented in order to establish the efficacy of it. This paper presents cognitive, contextual and scoring validity (reliability) evidence from the first and second versions of the test and investigates whether the modifications made after the first administration have had a positive effect on the quality of the test. The study concludes that although the changes made in the first version of the test strengthened the validity claims in terms of cognitive and contextual requirements, the reliability scores of the test worsened in the second version. This reminded us that although it is necessary to build the foundations of a test firmly by operationalizing the necessary contextual features and cognitive processes, this will not thoroughly guarantee the technical quality of the items. Scoring validity should be established carefully as well. This study exemplifies a thorough attempt in establishing the validity of a TSL test from multiple perspectives and aims to be an exemplary study for further test development in TSL.

Keywords

Cognitive validity, contextual validity, scoring validity, assessment of listening in Turkish as a second language

References

• ALTE, (2011). Manual for language test development and examining. Downloaded from https://www.alte.org/resources/Documents/ManualLanguageTest-Alte2011_EN.pdf
• Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
• Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.
• Borsboom, D., Mellenbergh, G. J., & Heerden, J. V. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071.
• Council of Europe, (2001). Common European framework of reference for languages: Learning, teaching and assessment. Cambridge: Cambridge University Press.
• EALTA guidelines for good practice in language testing and assessment. Downloaded from http://www.ealta.eu.org/documents/archive/guidelines/English.pdf
• Elliott, M., & Wilson, J. (2013). Context validity. In A. Geranpayeh & L. Taylor (Eds.), Examining listening: Research and practice in assessing second language listening, Studies in language testing, 35 (pp.152-241). Cambridge: Cambridge University Press.
• Field, J. (2013). Cognitive Validity. In A. Geranpayeh & L. Taylor (Eds.), Examining listening: Research and practice in assessing second language listening, Studies in language testing, 35 (pp.77-151). Cambridge: Cambridge University Press.
• Geranpayeh, A. (2013). Scoring validity. In A. Geranpayeh & L. Taylor (Eds.), Examining listening: Research and practice in assessing second language listening. Studies in language testing, 35 (pp.242-272). Cambridge: Cambridge University Press.
• Kane, M. T. (2013) Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement 50(1), 1–73.
• Khalifa, H., & Weir, C. J. (2009). Examining reading: research and practice in assessing second language reading, Studies in language testing, 29. Cambridge: UCLES/Cambridge University Press.
• Knoch, U., & Elder, C. (2013). A framework for validating post-entry language assessments (PELAs). Papers in Language Testing and Assessment, 2(2), 48- 66.
• McNamara, T. (2000). Language testing. Oxford: Oxford University Press.
• Messick, S. A. (1989). Validity. In R. L. Linn (Ed) Educational measurement 13-103. New York. American Council on Education. Mac Millian Publishing Company.
• Richards, J. C. (1983). Listening comprehension: Approach, design, procedure. TESOL quarterly, 17(2), 219-240.
• Tozlu, E. (2017). The development of a listening test for learners of Turkish as a foreign language (Unpublished master thesis). Boğaziçi University, İstanbul, Turkey.
• Trim, J. L. M. (2009). Breakthrough. Retrieved from https://www.coe.int/t/dg4/linguistic/Source/FinalBreakthrough%20specification_6Nov01.rtf
• Van Ek, J., & Trim, J. L. M. (1991a). Threshold 1990. Cambridge: Cambridge University Press.
• Van Ek, J., & Trim, J. L. M. (1991b). Waystage 1990. Cambridge: Cambridge University Press.
• Van Ek, J., & Trim, J. L. M. (2001). Vantage. Cambridge: Cambridge University Press.
• Walt, J. L., & Steyn, F. (2008). The validation of language tests. Stellenbosch Papers in Linguistics, 38, 191-204.
• Weir, C. J. (1993). Understanding and developing language tests. New York & Toronto: Prentice-Hall.
• Weir, C. J. (2005). Language testing and validation. Hampshire: Palgrave McMillan.
• Young, J.W., So, Y., & Ockey, G.J. (2013). Guidelines for best test development practices to ensure validity and fairness for international English language proficiency assessments. Educational Testing Service. https://www.ets.org/s/about/pdf/best_practices_ensure_validity_fairness_english_language_assessments.pdf