GENERALIZABILITY ANALYSIS OF THE READING SUBTEST IN AN ENGLISH DIAGNOSTIC TEST

Abstract

Ensuring measurement precision in diagnostic language assessment remains a critical challenge for educational institutions worldwide. This study employs Generalizability Theory (G-Theory) with a fully crossed person × item (p × i) design to evaluate the reliability of a reading comprehension subtest within a diagnostic English test used for university admission placement. Using R statistical software with the gtheory package, responses from 63 prospective university students across 15 multiple-choice reading items were analyzed through complementary G-study and D-study frameworks. The G-study decomposed total score variance into three components: persons (11.2%), items (7.0%), and residual error combining person-by-item interaction with random error (81.8%). This variance structure reveals significant psychometric limitations—the low person variance falls substantially short of the 40% benchmark for high-stakes decisions, while the dominant residual component indicates considerable measurement inconsistency. Reliability analysis yielded a generalizability coefficient (Eρ²) of 0.67 and dependability coefficient (Φ) of 0.65, both failing to meet conventional thresholds of 0.80 and 0.75 for consequential educational decisions. However, D-study simulations demonstrate a clear pathway for improvement: extending the test to 25–30 items would elevate both coefficients above acceptable standards while remaining feasible within institutional constraints. These findings indicate that the reading subtest, in its current 15-item form, does not meet minimal quality standards for high-stakes placement use and requires targeted revision. This study demonstrates that G-Theory provides an accessible yet rigorous framework for diagnostic test evaluation, offering evidence-based guidance for enhancing assessment quality even within resource-constrained institutional settings.

Keywords

• Generalizability theory
• Diagnostic assessment
• Reading test
• Reliability
• Educational measurement
• Decision study

İNGİLİZCE TANILAYICI TESTTE OKUMA ALTTESİNİN GENELLENEBİLİRLİK ANALİZİ

Abstract

Bu çalışma, üniversiteye girişte kullanılan bir İngilizce tanılayıcı testte yer alan okuma alt testinin güvenirliğini değerlendirmek için Genellenebilirlik Kuramı’nı (G-Kuramı) uygulamaktadır. Test üç bileşenden oluşmaktadır: 15 dinleme maddesi, 15 okuma maddesi ve 20 dil kullanımı maddesi. Bu çalışmada yalnızca okuma alt testine odaklanılmış ve 15 çoktan seçmeli maddeden oluşan bu alt teste yanıt veren 63 katılımcının verileri G-çalışması ve D-çalışması çerçevesinde analiz edilmiştir. G-çalışması; kişi (%11,2), madde (%7,0) ve kalıntı hataya (%81,8) atfedilen varyans bileşenlerini tahmin etmiş ve puan varyasyonunun büyük bir kısmının açıklanamayan faktörlerden kaynaklandığını ortaya koymuştur. D-çalışması, farklı madde sayısı senaryolarında güvenirlik katsayılarını (ρ ve φ) simüle etmiştir. 15 maddeyle elde edilen genellenebilirlik katsayısı (ρ²) 0,67 ve güvenilirlik katsayısı (φ) 0,65 olup, orta düzeyde bir güvenirliğe işaret etmektedir. Simülasyonlar, madde sayısının 30’a çıkarılması durumunda ρ² ve φ değerlerinin sırasıyla yaklaşık 0,80 ve 0,79’a yükselebileceğini göstermektedir. Bu bulgular, okuma alt testinin mevcut haliyle yüksek riskli kararlar almak için sınırlı bir kapasiteye sahip olduğunu göstermektedir. Ancak, madde sayısında yapılacak hedefli artışlar, puanların güvenilirliğini önemli ölçüde artırabilir. Bu çalışma, tanılayıcı değerlendirme tasarımında G-Kuramı’nın değerini vurgulamakta ve eğitim ölçme bağlamlarında test güvenirliğini artırmaya yönelik uygulanabilir öneriler sunmaktadır.

Keywords

• Genellenebilirlik kuramı
• Tanılayıcı değerlendirme
• Okuma testi
• Güvenirlik
• Eğitimde ölçme
• Karar çalışması

References

1. Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. Continuum.
2. American Educational Research Association, A. P. A. & N. C. on M. in E. (2014). Standards for educational and psychological testing. American Educational Research Association.
3. Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge University Press. https://doi.org/10.1017/CBO9780511667350
4. Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. In The Modern Language Journal (Issue 1). Oxford University Press. https://doi.org/10.2307/328718
5. Bolus, R. E., Hinofotis, F. B., & Bailey, K. M. (1982). An introduction to generalizability theory in second language research. Language Learning, 32(2), 245–258. https://doi.org/10.1111/j.1467-1770.1982.tb00970.x
6. Brennan, R. L. (2001). Generalizability theory. Springer. https://doi.org/10.1007/978-1-4757-3456-0
7. Brennan, R. L. (2011). Generalizability theory and classical test theory. Applied Measurement in Education, 24(1), 1–21. https://doi.org/10.1080/08957347.2011.532417
8. Brown, H. D., & Abeywickrama, P. (2010). Language assessment principles and classroom practices. Pearson Education.

9. Buck, G., Tatsuoka, K., & Kostin, I. (1997). The subskills of reading: Rule‐space analysis of a multiple‐choice test of second language reading comprehension. Language Learning, 47(3), 423–466. https://doi.org/10.1111/0023-8333.00016
10. Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. Routledge. https://doi.org/10.4324/9780203866948
11. Carr, N. T. (2011). Designing and analyzing language tests. Oxford University Press.
12. Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (2008). Building a validity argument for the test of English as a foreign language. Routledge.
13. Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. John Wiley & Sons. https://doi.org/10.2307/2286194
14. Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163.
15. Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (5th ed.). Prentice Hall.
16. Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates Publisher.
17. Grabe, W. (2009). Reading in a second language: Moving from theory to practice. Cambridge University Press.
18. Grabe, W., & Stoller, F. L. (2019). Teaching and researching reading (3rd ed.). Routledge. https://doi.org/10.4324/9781315726274
19. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65–110). Praeger.
20. Huang, J., & Whipple, P. B. (2023). Rater variability and reliability of constructed response questions in New York State high-stakes tests of English language arts and mathematics: Implications for educational assessment policy. Humanities and Social Sciences Communications, 10, Article 860. https://doi.org/10.1038/s41599-023-02385-4
21. Huebner, A., & Lucht, M. (2019). Generalizability Theory in R. Practical Assessment, Research, and Evaluation, 24(1). https://doi.org/10.7275/5065-GC10
22. Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
23. Kintsch, W. (2013). Revisiting the construction-integration model of text comprehension and its implications for instruction. In D. E. Alvermann, N. J. Unrau, & R. B. Ruddell (Eds.), Theoretical models and processes of reading (6th ed., pp. 807–839). International Reading Association.
24. Koda, K. (2005). Insights into second language reading: A cross-linguistic approach. Cambridge University Press.
25. Lakin, J. M., & Lai, E. R. (2012). Multigroup generalizability analysis of verbal, quantitative, and nonverbal ability tests for culturally and linguistically diverse students. Educational and Psychological Measurement, 72(1), 139-158. https://doi.org/10.1177/0013164411408074
26. Lee, Y. W. (2015). Diagnosing diagnostic language assessment. Language Testing, 32(3), 299–316. https://doi.org/10.1177/0265532214565387
27. Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263. https://doi.org/10.1080/15434300903079562
28. Liao, R. J. T. (2022). The use of generalizability theory in investigating the score dependability of classroom-based L2 reading assessment. Language Testing, 40(1), 86–106. https://doi.org/10.1177/02655322211070840
29. Marcoulides, G. A., & Ing, M. (2013). The use of generalizability theory in language assessment. The Companion to Language Assessment, 1207–1223. https://doi.org/10.1002/9781118411360
30. McNamara, D. S., & Magliano, J. (2009). Toward a comprehensive model of comprehension. Psychology of Learning and Motivation, 51, 297-384. https://doi.org/10.1016/S0079-7421(09)51009-2
31. Mislevy, R. J., Steinberg, L. s, & Almond, R. G. (2003). On the structure of educational assessments: CSE Technical Report (Report ED480557). Center for the Study of Evaluation, University of California, Los Angeles. https://eric.ed.gov/?id=ED480557
32. Nunnally, J., & Bernstein, I. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
33. Perfetti, C. A., Landi, N., & Oakhill, J. (2005). The acquisition of reading comprehension skill. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 227–247). Blackwell Publishing.
34. Perfetti, C., & Stafura, J. (2014). Word knowledge in a theory of reading comprehension. Scientific Studies of Reading, 18(1), 22–37. https://doi.org/10.1080/10888438.2013.827687
35. Rios, J., & Wells, C. (2014). Validity evidence based on internal structure. Psicothema, 26(1), 108–116. https://doi.org/10.7334/psicothema2013.260
36. Rupp, A. A., & Zumbo, B. D. (2006). Understanding parameter invariance in unidimensional IRT models. Educational and Psychological Measurement, 66(1), 63–84. https://doi.org/10.1177/0013164404273942
37. Sekretariat Direktorat Jenderal Pendidikan Tinggi, R. dan T. (2023). Buku statistik pendidikan tinggi 2023: Tabel 21–30. Pusat Data dan Teknologi Informasi Pendidikan Tinggi (PDDIKTI).
38. Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. SAGE Publications.
39. Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006). Reliability coefficients and generalizability theory. In Handbook of Statistics (Vol. 26, pp. 81–124). Elsevier. https://doi.org/10.1016/S0169-7161(06)26004-8
40. Xi, X. (2010). How do we go about investigating test fairness? Language Testing, 27(2), 147–170. https://doi.org/10.1177/0265532209349465