Face Validity Study of a Small-Scale Test in a Tertiary-Level Intensive EFL Program

Abstract

The assessment of language abilities play a crucial role in designing tests. In order to create a positive and motivating learning environment for test takers, one of the most important factors is the administration of tests with high levels of face validity as well as test taker cognizance of the abilities necessary for exam success. Therefore, this study examined the overlap in test takers’ and language instructors’ perceptions of the abilities being tested as well as test developers’ intentions in preparing exam items. The study was conducted at Gaziantep University Higher School of Foreign Languages with 38 test takers, seven language instructors, and four test designers. Data was collected by means of a response sheet on test takers’ perceptions of tests after being modified in accordance with the test to be analysed. The results revealed a moderate level of agreement between test designers and language instructors while a lower agreement percentage was obtained with test takers.

Keywords

• Face validity,small-scale tests,expert opinion,test taker perception

Üniversite İngilizce Hazırlık Sınıfında Küçük Ölçekli Bir Sınav için Görünüş Geçerliği Çalışması

Abstract

Dil becerilerinin ölçülmesi ve değerlendirilmesi eğitim öğretim çalışmaları kapsamında önemli bir alandır ve eğitim öğretim süreçlerini de şekillendiren önemli sonuçlara sahiptir. Sınava girecek öğrenciler için olumlu ve motive edici bir öğrenme ve sınav ortamı oluşturmak hem sınav görünüş geçerliği kavramının temeli hem de sınavı hazırlayanların amaçlarından bir tanesidir. Bu amaçla, bu çalışmada sınava giren öğrencilerin, sınavı hazırlayan ve derse giren öğretim elemanlarının sınavdaki maddelerin hazırlanma amaçları ile ilgili görüşleri araştırılmıştır. Bu çalışmada Gaziantep Üniversitesi Yabancı Diller Yüksekokulu İngilizce hazırlık sınıfında öğrenim gören 38 öğrenci, aynı birimden yedi öğretim elemanı ve dört sınav hazırlama komisyonu üyesi yer almıştır. Cevap kodlama formu ve bire bir görüşme yöntemleri kullanılarak nicel ve nitel veri toplanmıştır. Araştırma sonuçlarına göre sınav hazırlayanlar ile derse giren öğretim elemanlarının sınav sorularının amaçlarını eşleştirmede orta düzeyde bir uyuma sahip olduğu, öğrencilerde ise düşük seviyeli bir uyum olduğu belirlenmiştir.

Keywords

• Kapsam geçerlik,küçük ölçekli sınavlar,uzman görüşü,sınava giren öğrenci algısı

References

1. Barnett, L. M., Ridgers, N. D., Zask, A., & Salmon, J. (2015). Face validity and reliability of a pictorial instrument for assessing fundamental movement skill perceived competence in young children. Journal of Science and Medicine in Sport, 18, 98–102. https://doi.org/10.1016/j.jsams.2013.12.004
2. Brown, A. (1993). The role of test-taker feedback in the test development process: test- takers’ reactions to a tape-mediated test of proficiency in spoken Japanese. Language Testing, 10, 277-303.
3. Brown, HD, & Abeywickrama, P. (2010). Language assessment: Principles and classroom practices (2nd ed.). White Plains, NY: Pearson Education.
4. Brown, J. D. & Hudson, T. (2002). Criterion-referenced language testing. Cambridge: Cambridge University Press.
5. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82(1), 143–159. https://doi.org/10.1037/0021-9010.82.1.143
6. Davies, A, Brown, A, Elder, C, Hill, K, Lumley, T, & McNamara, T. (1999). Dictionary of language testing. Cambridge: Cambridge University Press.
7. Elder, C. (2007). OET-IELTS benchmarking study report. Australia: Language Testing Research Centre, University of Melbourne.
8. Grand, J. A., Ryan, A. M., Schmitt, N., Hmurovic, J. (2010). How Far Does Stereotype Threat Reach? The Potential Detriment of Face Validity in Cognitive Ability Testing. 24(1), 1-28. http://doi.org/10.1080/08959285.2010.518184

9. He, L., & Shi, L. (2008). ESL test takers’ perceptions and experiences of standardized English writing tests. Assessing Writing, 13(2), 130-149.
10. Hill, K. (1998). The effect of test-taker characteristics on reactions to and performance on an oral English proficiency test. In A.J. Kunnan (Ed.) Validation in language assessment (pp. 299-323). Mahwah, NJ: Lawrence Erlbaum.
11. Holden, R. R. (2010). Face Validity. The Corsini Encyclopedia of Psychology: John Wiley & Sons, Inc.
12. Hughes, A. (2003). Testing for language teachers. Cambridge: Cambridge University Press.
13. Kafer, K. L., & Hunter, M. (1997). On testing the face validity of planning/problem-solving tasks in a normal population. Journal of the International Neuropsychological Society, 3(2), 108–119.
14. Kim, H., & Elder, C. (2015). Interrogating the construct of aviation English : Feedback from test takers in Korea. Language Testing, 32(2), 129–149. http://doi.org/10.1177/0265532214544394
15. Kelecioğlu, H., & Şahin, S. G. (2014). Geçmişten Günümüze Geçerlik. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 5(2), 1–11.
16. Knoch, U. (2014). Using subject specialist to validate an ESP rating scale: The case of the International Civil Aviation Organization (ICAO) rating scale. English for Specific Purposes, 33, 77–86.
17. MacLellan, E. (2001). Assessment for learning: The differing perceptions of tutors and test takers. Assessment & Evaluation in Higher Education, 26(4), 307–318.
18. Nevo, B. (1985). Face Validity Revisited. Journal of Educational Measurement, 22(4), 287–293. https://doi.org/10.1111/j.1745-3984.1985.tb01065.x
19. Sato, T., & Ikeda, N. (2015). Test-taker perception of what test items measure: a potential impact of face validity on student learning. Language Testing in Asia, 5(10). http://doi.org/10.1186/s40468-015-0019-z
20. Scouller, K. M., & Prosser, M. (1994). Students' experiences in studying for multiple choice question examinations. Studies in Higher Education, 19(3), 267-279.
21. Schmitt, N. (2002). Do reactions to tests produce changes in the construct measured? Multivariate Behavioral Research, 37(1), 105–126. http://doi.org/10.1207/S15327906MBR3701
22. Shohamy, E. (2001). Democratic assessment as an alternative. Language Testing, 18(4), 373-391.
23. Paker, T. (2013). The backwash effect of the test items in the achievement exams in preparatory classes. Procedia-Social and Behavioral Sciences, 70, 1463–1471.http://doi.org/10.1016/j.sbspro.2013.01.212
24. Teemant, A. (2010). ESL student perspectives on university classroom testing practices. Journal of the Scholarship of Teaching and Learning, 10(3), 89-105.
25. Tsaia, Y., & Tsou, C.-H. (2009). A standardised English language proficiency test as the graduation benchmark: Student perspectives in higher education. Assessment in Education: Principles,Policy & Practice, 16(3), 319-330.
26. Watkins, D., Dahlin, B. O., & Ekholm, M. (2005). Awareness of the backwash effect of assessment : A phenomenographic study of the views of Hong Kong and Swedish lecturers. Instructional Science, (33), 283–309. http://doi.org/10.1007/s11251-005-3002-8
27. Xie, Q. (2011). Is test taker perception of assessment related to construct validity? International Journal of Testing, 11, 324–348. http://doi.org/10.1080/15305058.2011.589018
28. Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.